Program with Spark

Canonical programming guide for the Spark language on SoapBox. Statement reference: Language reference.

Primary today: CLI ./spark (this guide). Verified IDE language ops: ide new|open|save|run|buffer|ask|show and ide keys / ide key — IDE status. Editor paint is a wire + PPM (gutter/glyphs + AI strip; make test-ide-paint), not a .spark statement. ide show reuses real spark-engine-show. See verified ops below.

Rule: ops and flags below match asm/ dispatch, shipped examples/*.spark, and commands verified under --dry-run (or companion --dry) on this tree.

Repos: language SoT = /home/mike/workspaces/spark. Browser product host = /home/mike/workspaces/spark-browser (Qt shim). Engine-B asm examples exist in-repo (see §10); they are not a substitute for the language CLI workflow.

1. What Spark is

You write .spark files. Today they run on the shipped ./spark ELF GAS scaffold (asm/). Authoring SoT DECIDED (A+B+C): self-host destination (A), thin C bootstrap (bootstrap/), Spark-native assembler (sparkasm/). Do not treat GAS as forever SoT; do not start a Rust/Python VM from IDE work. See Self-host path.

Docs cover proven ops and fixture HTML examples you can dry-run today.

Live HTTP for ask is a companion (./spark-ask-http) forked only under --live. Dry-run does not dial the network for ask.

This is not Apache Spark / Databricks.

2. Build

cd /home/mike/workspaces/spark
        make                 # ./spark + companions
        ./spark --version    # e.g. spark 0.6.0 (x86_64 asm + companions)
        make machine-proof   # file(1) + objdump of _start
        

If make fails on asm/engine_*.s while parallel engine work is in flight, that is an engine-lane assemble/link issue — not a reason to invent a different language runtime. Re-run make when those objects link; language examples below were verified when ./spark was present.

3. Run: dry vs live

Verified usage from bare ./spark:

Usage: spark --dry-run [--allow-net] [--allow-net-capture] <file.spark>
               spark --live <file.spark>  # ask → AI_GATEWAY_URL
               spark --live --pstn-live <file>  # PSTN off unless SPARK_PSTN=1
               spark --version
        

There is no ./spark ide ELF subcommand. IDE core is language ops (ide open|save|run|…) — IDE status, examples/ide_hello.spark.

./spark --dry-run examples/hello.spark
        ./spark --dry-run examples/ide_hello.spark

        export AI_GATEWAY_URL=http://127.0.0.1:4000
        export OPENAI_API_KEY=…          # sk-bf-* ; never commit
        ./spark --live examples/ask_live.spark
        

4. First program

File: examples/hello.spark (verified rc=0 under --dry-run):

model code

        ask "Explain gravity in one sentence" -> text

        print text
        

./spark --dry-run examples/hello.spark
        

Lexical rules (from LANGUAGE.md / working examples):

• Comments: # to end of line
• Strings: "..."
• -> name binds a result
• | may prefix a pipeline step

Optional spark.toml documents defaults; the dry VM does not parse it for aliases — use a model statement in the .spark file.

5. Syntax (verified examples)

Each subsection names the file that was dry-run successfully.

5.1 model + ask

examples/hello.spark, examples/ask_live.spark

model code          # or fast | best (gateway aliases)

        ask "Explain gravity in one sentence" -> text
        print text
        

Live ask needs --live + ./spark-ask-http + env — ASK_LIVE.md (ASK_LIVE.md in repo).

5.2 ask probe / gateway probe

examples/ask_probe.spark, examples/gateway_probe.spark
(requires companion ./spark-ask-probe; dry prints Infisical readiness JSON, no public HTTP)

ask probe -> info
        gateway probe -> info
        

./spark --dry-run examples/ask_probe.spark
        ./spark-ask-probe --dry
        

Live public probe / HTTP 401 → credential unavailable (exit non-zero); see ASK_LIVE.md.

5.3 classify

examples/classify_intent.spark

classify Intent { support, sales, spam }
          from "My washer is broken and I need help"
          min_confidence 0.7
          -> intent

        classify multi Tags { support, sales, spam }
          from "Please help me buy a card, what is the price?"
          -> tags
        

5.4 extract

examples/extract_person.spark

extract Person {
          name: string
          age: int
        } from "Ada Lovelace was born in 1815" -> person
        

5.5 pipeline

examples/pipeline_translate.spark

let doc "Laundry machines need regular cleaning and balance checks."

        pipeline {
          ask "Summarize: {doc}" -> summary
          | ask "Translate to Spanish: {summary}" -> es
        }
        

5.6 tool / with tools

examples/tool_agent.spark

tool weather(city: string) -> string {
          "stub:local"
        }

        with tools [weather] {
          ask "What's the weather hint for Des Moines?" -> answer
        }
        

Fail-loud cases live under examples/neg/ (e.g. with_no_tool.spark).

5.7 review / builder / implement

examples/review_builder.spark, examples/ide.spark
(dry-run writes out/program.spark; not the product IDE)

review path "examples/fixtures/sample.js" -> report
        review url "file://examples/fixtures/sample.js" -> report_url
        review text "function x(){ eval(y); }" -> report_text

        builder prefer lower request "add classify Intent and wire voice turn" -> patch
        implement patch into "out/program.spark"
        

Remote http(s) review needs --allow-net. Never evaluates web JS.

5.7b IDE core (ide)

Verified (examples/ide_hello.spark, examples/ide_ask.spark, examples/ide_show.spark, make test, ask ship dc54d92, this tree rc=0):

ide new
        ide open "examples/hello.spark" -> opened
        ide buffer -> shown
        ide save "out/ide/ide_hello_saved.spark" -> saved
        ide run -> ran
        ide ask -> reply
        ide show -> shown
        

./spark --dry-run examples/ide_ask.spark
        # dry → fixture reply (e.g. Gravity); live → spark-ask-http

        ./spark --dry-run examples/ide_show.spark
        # dry → validate editor.ppm; live → spark-engine-show

        ./spark --dry-run examples/ide_run_show.spark
        # open → run (child) → show (engine validate + show.json)

        ./spark --dry-run examples/ide_ask_show.spark
        # open → ask (AI strip) → show (engine validate + show.json)

        ./spark --dry-run examples/ide_save_reopen.spark
        # open → save → reopen; saved file sha256/size == hello.spark
        

Display window = engine companion. Details: IDE status. Reports: reports/spark-ide-ask-20260831.md, reports/spark-ide-show-20260831.md, reports/spark-ide-run-show-20260831.md, reports/spark-ide-ask-show-20260831.md, reports/spark-ide-save-reopen-20260831.md.

5.7c IDE keymap (ide keys / ide key)

Verified (examples/ide_keys.spark, commit 51fbe8f + show polish, save token proof examples/ide_keys_save.spark, run token proof examples/ide_keys_run.spark, this tree rc=0). Asm: asm/ide_keys.s. Script fixtures: examples/fixtures/ide/cmds.txt, cmds_save.txt (s → save), cmds_run.txt (r → run).

ide keys "examples/fixtures/ide/cmds.txt"

        ide key open "examples/hello.spark"
        ide key show
        ide key save
        ide key run
        ide key quit
        

./spark --dry-run examples/ide_keys.spark
        # prints {"op":"ide.keys",…}; appends out/ide/keys_trace.jsonl

        ./spark --dry-run examples/ide_keys_save.spark
        # script `s` → {"cmd":"save","path":"examples/hello.spark"}; dry no write

        ./spark --dry-run examples/ide_keys_run.spark
        # script `r` → {"cmd":"run","path":"examples/hello.spark"}; dry no fork
        

Dry-run: traces only for save/run (no write / no fork). Open still does real open+read and paints editor.ppm. Show dry-validates via engine_window_show. No mouse GUI. Do not invent key tokens.

5.7d Editor paint (PPM — not a language op)

Verified (make test-ide-paint, status strip b029a4a + dirty * after ide new + AI panel on dc54d92, PASS this tree). There is no ide paint statement in .spark. Paint is asm ABI: core exports ide_buf / ide_buf_len / ide_dirty and calls ide_paint_bind after new/open/save; ide ask calls ide_ai_set then rewrites PPM. Paint owns ide_cursor / ide_status_set / ide_ai_set and writes out/ide/editor.ppm (top status path[+*] + gutter + glyphs + cursor + bottom AI strip). ide new → dirty *; open/save clear. Keymap n→new not proven. Example: examples/ide_dirty_status.spark.

make test-ide-paint
        # → {"op":"ide.paint","ppm":"out/ide/editor.ppm",…}
        # → PASS ide-paint-ppm … / ide_status_strip_ppm
        ./spark --dry-run examples/ide_dirty_status.spark
        # → dirty true on new; status_dirty.txt ends *; save clears
        

Report: reports/spark-ide-status-strip-20260831.md, reports/spark-ide-dirty-status-20260831.md.

5.8 Voice

examples/voice_turn.spark (and VOICE.md examples)

voice {
          listen -> user
          classify Intent { support, sales } from user -> intent
          ask "Reply helpfully to: {user}" -> reply
          speak reply -> "out.wav"
        }
        

Also present: voice_reviewer.spark, voice_coder.spark, voice_copy.spark, voice_model.spark, voice_pstn.spark, voice_live.spark. PSTN stays gated off unless --pstn-live + SPARK_PSTN=1.

5.9 Browser / MITM (language driver)

Canonical dry entry: examples/browser_main.spark

browser run "examples/browser_main.spark" -> session
        browser goto "https://example.com/" -> page
        mitm enable -> mitm_session
        mitm filter "example\\.com" -> filt
        mitm har export -> har
        

Also dry examples: browser_ca.spark, browser_h2.spark, browser_quic.spark, browser_cdp.spark, browser_mitm.spark, browser_show.spark (PPM show; dry display:false; live X11 via ./spark-engine-show — see §10 Live X11 show).

browser gui requires --live (forks ./spark-browser-host). Dry browser gui fails by design (examples/neg/browser_gui_dry.spark).

Do not document python3 -m spark_browser run as the product entry.

5.10 Encrypt gateway

examples/encrypt_gateway.spark (needs ./spark-enc-gateway)

crypto keygen -> key
        encrypt gateway enable key
        gateway encrypt on
        encrypt seal text "…" -> blob
        encrypt open blob -> plain
        ask "…" -> reply
        gateway encrypt off
        

5.11 Network / binary / CUDA / OS / model

6. Layout

examples/*.spark     # programs
        docs/                # this guide + LANGUAGE.md + …
        asm/*.s              # VM (do not thrash engine_* from app docs)
        tools/               # companions (ask-http, review-url, …)
        templates/           # os / browser scaffolds
        out/                 # implement / HAR / decompile / engine artifacts
        tests/               # dry harness
        spark.toml           # documented defaults (not VM-parsed for aliases)
        spark.code-workspace # Cursor workspace (IDE.md)
        

7. Debugging

Negative corpus: examples/neg/ (tests expect non-zero exit + error text).

Errors / exit codes (verified)

Do not assume every stderr line means a non-zero process exit without checking $? (pipes hide the VM’s code unless set -o pipefail).

8. Tests

make test
        make test-examples
        make test-e2e-browser
        make test-ide-paint   # PPM paint wire (not a .spark op)
        

make test stays offline for ask/vendor speech. If ./spark disappears mid-suite while engine objects are relinking, re-run make then make test.

9. Hardware / asm

• Users program in Spark. The VM is asm→ELF.
• Do not edit asm/engine_*.s from an app/docs lane while engine agents work that tree.
• make machine-proof shows ELF64 + _start disassembly when the binary links.

10. Engine B (verified ops only)

Documented here only after dry-run (or gated fail) on this tree.

Phase-1 <script> → engine_js_eval. HTTPS uses OpenSSL BIO companion ./spark-engine-fetch-tls.

Dry pipeline (fetch→parse→css→layout→paint→show)

examples/engine_pipeline.spark (verified on this tree):

engine fetch "file://engine/fixtures/style_basic.html" -> body
        engine parse "out/engine/body.bin" -> dom
        engine css attach -> styles
        engine layout -> boxes
        engine paint boxes -> ppm
        engine show "out/engine/pipeline.ppm" -> shown
        

./spark --dry-run examples/engine_pipeline.spark
        # → out/engine/pipeline.ppm + out/browser/show.json
        #   {"op":"show","mode":"dry-run","display":false,…}
        

Table pipeline (real): examples/engine_pipeline_table.spark on table_demo.html — cell borders + cell text (layout.table / cell_text). Also examples/engine_fetch_parse_layout.spark. Proofs: make test-engine-pipeline-table, make test-engine-layout, make test-engine-fetch-parse-layout.

CSS padding (real): shorthand padding px on block/cell boxes (e313b21); border-width px on table cells (60e2b44); display subset includes table/table-row/table-cell, plus padding and border-width in px.

Artifacts: out/engine/body.bin, out/browser/engine/dom.json, out/browser/engine/css.json, out/engine/pipeline.ppm, out/browser/show.json. Layout without prior parse fail closed (examples/neg/engine_layout_no_dom.spark).

Live X11 show (spark-engine-show)

Same .spark file; --live forks the existing companion (no Qt):

make spark-engine-show
        ./spark --live examples/engine_pipeline.spark
        # companion: ./spark-engine-show --ppm out/engine/pipeline.ppm --hold 2000
        # show.json → mode:live display:true path:out/engine/pipeline.ppm
        

Also: ./spark --live examples/browser_show.spark (fixture PPM) and ./spark --live examples/browser_engine_render.spark (render→same pipeline.ppm). Needs DISPLAY. Longer view:

./spark-engine-show --ppm out/engine/pipeline.ppm --hold 5000
        ./spark-engine-show --dry --ppm out/engine/pipeline.ppm   # no X11
        

Dry make test never forks X11. Report: reports/spark-engine-show-live-20260831.md.

engine parse → out/browser/engine/dom.json

examples/engine_parse.spark (verified rc=0):

engine parse "engine/fixtures/hello.html" -> dom
        

Writes DOM JSON under out/browser/engine/dom.json ("op":"engine.parse", "engine":"spark-asm-html", node list).

engine fetch

engine fetch "file://examples/fixtures/engine/sample.html" -> body
        # http/https with --allow-net:
        # ./spark --dry-run --allow-net your_fetch.spark
        

HTTPS is not TLS-in-asm / not Python — companion only (27252f4). PASS: engine_fetch_tls_companion_* + engine_fetch_https_openssl_bio.

js phase-1 only

Verified: examples/js_phase1.spark, browser/engine/js.spark.

js eval "1+1" -> sum
        js eval "'a'+'b'" -> cat
        js eval "-1" -> neg
        js eval "var x = 1; x+2" -> vsum
        js run "console.log(1+1)" -> log
        js console -> buf          # dump console buffer (browser/engine/js.spark)
        js selftest -> ok
        

Runtime prints phase1 numbers/strings/+/unary- /var-num/console.log. Artifacts: out/engine/js_result.txt, out/engine/js_console.txt.

DOM hook (real): engine parse walks <script> text children and calls engine_js_eval (same phase-1 slice). Proof: engine/fixtures/hello.html (1+1), engine/fixtures/script_log.html, ./engine/tests/js_script_dom.sh.

Unary -: prefix on a number primary (-1, -1+3, --1). Binary - and -'str' fail loud.

var number assign: var x = 1 / var x = 1; x+2 (RHS must be number). var x = 'a', undeclared x, let/const fail loud.

Phase-1 slice: numbers, strings, +, unary -, var num, and console.log.

Engine B pixels are the product render path. See Self-host path.

11. What Spark includes today

Real surface area you can dry-run now:

model / ask / classify / extract / pipeline / tool / review→builder→implement / voice / gateway encrypt / verified ide ops / engine B pipeline. A+B+C: Self-host path, bootstrap/README.md, sparkasm/README.md, selfhost/README.md.

12. Cheat sheet

make && ./spark --version
        ./spark --dry-run examples/hello.spark
        ./spark --dry-run examples/ide_hello.spark
        ./spark --dry-run examples/ide_ask_show.spark
        ./spark --dry-run examples/ide_save_reopen.spark
        ./spark --dry-run examples/engine_pipeline.spark
        ./spark --dry-run examples/engine_pipeline_table.spark
        ./spark --dry-run examples/js_phase1.spark
        make test-bootstrap
        make -C sparkasm test
        make test-selfhost-lex
        # make test-ide-paint   # PPM + status strip (not a .spark op)
        # optional interim editor: make ide  (Cursor host — see IDE.md)
        

Next: IDE status (verified ide ops; CLI stays primary).