Spark bytecode ISA (Phase 0)

Destination execution SoT for the self-host lane: .sparkbc runs on bootstrap/bc_vm.c (spark-bootstrap --run-bc). Tree-walk spark_vm_run_file remains a legacy bridge until later phases delete it.

This is not JVM, WASM, or CPython bytecode. Magic is Spark-only.

Source of truth (no inventing)

Every opcode and stdout string traces to:

1. LANGUAGE.md — statement forms
2. ../examples/hello.spark — model / ask / print
3. GAS + C bootstrap dry-run stdout (same model/ask/→/print lines)
4. Existing dry fixtures in bootstrap/vm.c / bootstrap/dry_ask.c spark_pick_ask_reply (copied from pick_ask_reply; do not invent new reply prose)

No inventing. If a form is not in LANGUAGE.md, not in examples/hello.spark, and not an existing dry helper string: do not add an opcode, operand, or fixture. Phase 0 is only the LANGUAGE statement starts already wired for hello, plus a VM terminator.

File layout (little-endian, packed, no padding)

offset 0 : 4 bytes magic 'S' 'P' 'B' 'C'
        offset 4 : 1 byte version 0x01
         : u16 nstrings
         : nstrings × { u16 nbytes; nbytes UTF-8 bytes }
         : u16 nconsts
         : nconsts × { u8 kind; u16 payload }
         : u32 ncode
         : ncode bytes (opcode + operands)
        

Version must be 0x01. Other versions fail loud. Bad magic / truncated section / unknown opcode / unknown const kind / out-of-range index: fail loud, non-zero exit, no silent skip.

String pool

Quoted strings and identifiers from LANGUAGE / hello. No NUL in the file; the loader appends '\0' for C. Phase 0 hello strings are exactly:

Constant pool

Phase 0 kind only:

No other kinds. Code operands are u16 constant-pool indices. A STR const resolves to that string. hello.sparkbc uses three STR consts (0→"code", 1→prompt, 2→"text").

Code section

Stream of u8 opcode plus opcode-specific u16le const indices. HALT has no operands. Execution starts at byte 0.

Stack / frame model

Frame (used in Phase 0):

• model_alias — LANGUAGE model <alias> / hello model code
• named bindings — LANGUAGE let / -> bind; hello text
• last — last ASK/LET value (same role as bootstrap last_val)

Operand stack: reserved. Phase 0 hello never pushes or pops. No stack opcodes this phase (would be inventing).

Opcode table (Phase 0 + Phase 4)

Mapped 1:1 to LANGUAGE.md statement starts. Voice merge keeps 0x06 EXTRACT, 0x07 PIPELINE, 0x09 LISTEN, 0x0a SPEAK, 0x0d VOICE; engine/IDE 0x0e–0x1a; review/browser/mitm 0x1b–0x1f. Tool/with use 0x20–0x22 (after mitm).

TRAIN / STEP / TRAIN_STATUS are the training program in the binary. They encode job submit, one dry loop tick, and status — dumpable hex + mnemonics. Dry fixture via bootstrap/dry_train.c (JSON + ARTIFACT marker). Execute with ./spark-bootstrap --run-bc or ./spark --run-bc (GAS thin fork → bootstrap bc_vm). GAS --compile wraps bootstrap emit; C lowering remains SoT. GAS does not reimplement .sparkbc — use bootstrap --compile. Emitting TRAIN/STEP ≠ a trained model. backend is parsed and skipped (HTTP companion); not an operand.

EXPECT want is a literal, or @fixture:path when the source used fixture "path". Missing binding / missing fixture / mismatch → non-zero exit. Shared validator: bootstrap/dry_expect.c.

EMBED / RETRIEVE dry replies come from bootstrap/dry_rag.c (same fixtures as examples/fixtures/rag/). Live GAS forks ./spark-rag-http.

GAS-first (no opcode yet): http get / http post — dry fixture files via ./spark-http --dry (or bootstrap dry_http.c); live forks ./spark-http --live. See LANGUAGE.md. Do not invent BC operands until an opcode lands.

In SPARK_BC: model train / model build → 0x26 TRAIN; model step → 0x28 STEP; model status → 0x27 TRAIN_STATUS. See above. Companion ./spark-train-http stays the live HTTP job path. Emitting the opcode is the training program, not trained weights.

GAS-first (no opcode yet): model analyze / compare / improve / plan run in asm/model_ops.s. See MODEL_TRAINING.md.

LET is in the table so the ISA is not hello-only (LANGUAGE.md let / print / set). hello.sparkbc does not emit LET. Handler matches bootstrap/vm.c op_let (print + bind + last). set is GAS-only in bootstrap — not an opcode here.

HALT ends the program. Documented as VM-only; not a .spark keyword.

CLASSIFY (Phase 4 first family)

LANGUAGE form in selfhost/fixtures/classify_dry.spark (one line):

classify Intent { support, sales, spam } from "…" min_confidence 0.7 -> intent
        classify multi Tags { support, sales, spam } from "…" -> tags
        

Encoded operands (proven by dry-run SoT): two u16 const indices, same shape as ASK:

1. from-text — the from STRING. GAS / vm.c op_classify uses only extract_quote of that string.
2. bind — IDENT after ->. Bound value is the dry JSON.

Not encoded (same as vm.c): schema name (Intent / Tags), multi, { support, sales, spam } labels, min_confidence. C compile still parses those tokens from classify_dry and discards them. Do not invent a label-list operand.

Dry JSON is spark_pick_classify / vm.c pick_classify (copied to bootstrap/dry_classify.c): buy/price → sales; broken/help/ support → support; spam → spam; else support. Strings are the existing DRY_SALES / DRY_SUPPORT / DRY_SPAM fixtures. No new prose.

GAS classify_dry body (compare SoT; banner may differ):

[model] fast
        [classify] My account is locked
         → {"label":"support","confidence":0.91,"reasons":["dry-run"]}
        [classify] Please help me buy a card
         → {"label":"sales","confidence":0.88,"reasons":["dry-run"]}
        [print] {"label":"support","confidence":0.91,"reasons":["dry-run"]}
        [print] {"label":"sales","confidence":0.88,"reasons":["dry-run"]}
        [spark] ok
        

EXTRACT

LANGUAGE form in selfhost/fixtures/extract_dry.spark:

extract Person { name: string, age: int }
         from "…" fixture "examples/fixtures/extract/person.json" -> person
        

Encoded operands: three u16 const indices.

1. schema — the declaration re-serialized to one line, extract NAME { f: type, g?: type }. The compiler joins the block's tokens; source newlines and commas both separate fields, so the encoded text is canonical regardless of layout.
2. fixture — the fixture STRING. Required: a compile without one is an error, because dry-run has no other source of values.
3. bind — IDENT after ->.

The schema travels as text, not a packed field table, so bootstrap/dry_extract.c is the single validator shared by the GAS VM, the bootstrap interpreter, and this VM. All three therefore accept and reject exactly the same inputs — run_sparkbc.sh compares bodies byte-for-byte.

Not encoded: the from STRING. It is the prompt a live model would receive, and live extract is not wired; encoding it would imply a capability that does not exist.

op_extract (bc_vm.c) parses the schema, loads the fixture from disk, validates, then prints [extract] <json> and binds. A missing fixture or a failed field check returns non-zero — it does not fall back to a built-in value.

TOOL / WITH / WITH_END (Phase 4 second family)

LANGUAGE form in examples/tool_agent.spark (same bytes as bootstrap/fixtures/tool_agent.spark):

tool weather(city: string) -> string { "stub:local" }
        with tools [weather] { ask "…" -> answer }
        print answer
        

Cited vm.c before encoding:

• op_tool (~588) — name before (, print [tool] registered %s. Signature / { "stub:local" } ignored.
• op_with (~608) — requires tools, prior registration, [list]; copies bytes between [ and ] into tools_scope; sets tools_active = 1; prints the existing JSON line.
• } (~1559–1562) — tools_active = 0; no stdout. That is WITH_END
• op_ask (~468–471) — when tools_active && tool_reg_len: reply is the existing [tool:NAME] stub:local (not pick_ask_reply).

Encoded operands (proven by that fixture + dry-run):

1. TOOL — one u16 const: tool name (weather).
2. WITH — one u16 const: list text between [ and ] (weather for [weather]).
3. WITH_END — no operands.

Not encoded (same as vm.c): (city: string), -> string, { "stub:local" } body. C compile parses those tokens and discards them. Do not invent a signature operand.

GAS tool_agent body (compare SoT; banner may differ):

[model] code
        [tool] registered weather
        [with] {"op":"with_tools","tools":"weather","active":true}
        [ask] What's the weather hint for Springfield?
         → [tool:weather] stub:local
        [print] [tool:weather] stub:local
        [spark] ok
        

Dry-run semantics (stdout contract)

--run-bc is dry-run only (no network). Lines for model / ask / arrow / print must be byte-identical to GAS ./spark --dry-run examples/hello.spark and bootstrap ./spark-bootstrap --dry-run examples/hello.spark.

GAS hello:

[spark] dry-run via assembly VM (machine code)
        [model] code
        [ask] Explain gravity in one sentence
         → Gravity pulls masses together.
        [print] Gravity pulls masses together.
        [spark] ok
        

Bytecode banner (allowed distinct line):

[spark] dry-run via bytecode VM
        

then the same [model] / [ask] / → / [print] / [spark] ok.

ASK reply: spark_pick_ask_reply — contains_ci "gravity" → "Gravity pulls masses together." (existing fixture).

PRINT of a bound name: [print] <value>\n (vm.c op_print). Unbound name prints the ident (same as vm.c). MODEL: [model] <alias>\n.

hello.sparkbc hex / layout

Packer (encoding, not a compiler): bootstrap/bc_pack_hello.c + bootstrap/bc_write.c. Reconstruct by packing the three strings and four ops above, or make spark-bc-pack-hello and write the golden. Phase 3 C --compile must emit the same bytes for hello.

Packed size of the golden: 79 bytes (magic+ver 5 + nstrings 2 + strings 2+4 + 2+31 + 2+4 + nconsts 2 + 3×3 + ncode 4 + 12 code). Code section starts at file offset 67.

Code bytes:

01 00 00 MODEL const0
        02 01 00 02 00 ASK const1 const2
        03 02 00 PRINT const2
        00 HALT
        

Builder (Spark-created binary + weights)

SPARK_BC is the orchestration ISA — not neural weights. Full engineer reproduction (programs, commands, sha256 table, GAS --compile / --run-bc, model lab, Pages deploy): SPARK_BC Builder.

Spark compiling Spark: selfhost/compile.spark → docs/examples/spark-self.sparkbc (sha256 a33f3232752c2fdd75db728773cb589a4eaf8e0bb70ac2539d2519b2dc7df6e8). Train slice: selfhost/compile_train.spark → docs/examples/spark-selfhost-train.sparkbc (TRAIN / TRAIN_STATUS; sha256 0b524a96338e13c75efd431853ca33cc8ef227483d3ead0fbb56342da23f9d2b). Builder: examples/spark_builder.spark → docs/examples/spark-builder.sparkbc (sha256 e89b27c86618cbbb87d2d59209fb5c0450c0513105c88c622de367f3a8e29f9a). STEP proof: examples/spark_train_step.spark → docs/examples/spark-train-step.sparkbc (TRAIN → STEP 0x28 → TRAIN_STATUS; sha256 d08925b52bf8c840de626c9cfec619d4dbae5a674b94bb8c7c5837eb1ac64551). Dumps: examples/spark-builder-bc.txt, examples/spark-self-bc.txt, examples/spark-train-step-bc.txt. Init weights from those bytes: docs/examples/spark-self.init.safetensors (sha256 60b9b7297cb5e2d8362702144a9d9c15487e65dd11783ba7d499499b500198cf). Focused e2e: make sparkbc-e2e (compile → dump TRAIN/STEP → --run-bc → assert ARTIFACT; STEP = multi-pass CPU SGD). STEP weights + loss drop via make test-sparkbc. Factory page: SPARK_BC Builder. TRAIN accept is still a dry marker. STEP applies multi-pass CPU SGD (trained=true / not_sgd=false only after grads; checkpoint.json loss curve). Measurement only. ./spark --compile and ./spark --run-bc thin-wrap bootstrap (C lowering / bc_vm remain SoT).

Out of scope (do not add)

include, set, live ask. Phase 3 C --compile lowers hello/mini. Phase 4 adds classify, tool/with, extract, pipeline, voice, engine/IDE, review/browser/mitm per table above. Spark-hosted compiler is still out of scope. Later phases may grow the ISA from LANGUAGE.md only.