AI model aspects — sensory mapping for agentic Spark

Engineer map of the Spark / SparkLang AI model as a whole system: how text, tools, memory, and optional senses plug into bytecode + weights + serve. Not marketing. Not God/Loom metaphors.

Eval scores are measurement only, not marketing wins. Prefer CPU or a consumer GPU for opt-in large local work.

Factory hub: Factory hub. Knowledge hive: Knowledge (agents/tools · multimodal · eval). Voice surface: VOICE.md. Voice easy train: Voice easy. Voice ask (dump Q&A): VOICE_ASK.md. Methods vs OpenBin: Methods vs OpenBin. Coder: Spark coder. Weights: Weight gallery. Architecture: Architecture. Attention / forward: Attention / forward. Live ask: ASK_LIVE.md. Diagrams: Diagrams.

Forge nav: this page lives under Forge → Model aspects on sparklang.dev. Sibling Forge entries: Diagrams, Builder, Build models, Train loop, Spark coder, Weight gallery, AI models, Voice / STT / TTS.

Category map

Nav: Forge → Models → Model aspects (with Senses / Train siblings).

Why sensory mapping matters (hero)

Agentic systems are not “a chat box with a bigger context window.” They are closed loops: perceive → decide → act → speak → perceive again. If you only document the LLM forward pass, you hide the failure modes that dominate real agents:

Sensory mapping is the contract between plain language (“ears / eyes / speaking / thinking / behaviors”) and engineering (STT sidecar, vision stub, TTS PCM, tiny CPU forward, .spark policies). It is how Spark stays a local SoT language instead of a vague “AI platform” claim.

For agentic AI specifically:

1. Each sense is a boundary. Fail-closed net gates (SPARK_STT_NET, SPARK_TTS_NET, inventable-IDK on ask) live at the sense edges — not buried in marketing copy.
2. Behaviors own the loop. Tools and reply packs are first-class, not afterthoughts bolted onto logits.
3. Tiny vs large is explicit. CI stays tiny; large is opt-in on a consumer GPU or CPU.
4. Comparison without cloning. OpenBin Ask / production phone voice stacks solve different jobs; Spark documents its loop and refuses to fake theirs.

This page keeps the summary status table (below), then expands ~100× into diagrams, how-tos, opcode/CLI maps, gaps, and roadmap.

Sensory mapping (summary table)

Jump: Ears · Eyes · Speaking · Thinking · Behaviors · System diagram · Comparison · How-to lab · Capability status.

Status table (engineering aspects)

System diagram — ears → brain → voice ↔ tools

flowchart LR
         EAR["Ears — listen / STT
spark-stt-tts"]
         TOK["Tokenizer / bindings"]
         BRN["Thinking — tiny CPU forward
embed → attention → MLP → lm_head"]
         TOOL["Tools / behaviors
classify · ask · reply pack"]
         VOC["Speaking — speak / TTS
PCM or SPARK_TTS_*"]
         EAR --> TOK --> BRN
         BRN --> TOOL
         TOOL --> VOC
         VOC -.->|next turn transcript| EAR
         TOOL -.->|tool result text| BRN
         EYE["Eyes — vision
not available"] -.->|not wired| BRN

Loop reading (plain): ears (or typed text) become tokens → thinking produces logits / ask text → behaviors pick tools, SoT, or abstain → speaking (or print) leaves the system → the next turn may listen again. Eyes stay dashed: planned contract only.

Ears — STT / audio in

Plain language

Ears turn sound into text the rest of Spark can bind (->) and pipe (|). Without ears, agent demos stay keyboard-only.

Engineering stack

Dataflow

flowchart TB
         WAV["WAV file or arecord mic"]
         SIDE["sidecar .txt / .intent.txt"]
         CMD["SPARK_STT_CMD (%i → stdout)"]
         WH["openai-whisper tiny.en"]
         HTTP["HTTP STT — only if SPARK_STT_NET=1"]
         BIND["VM binds transcript -> var"]
         WAV --> SIDE
         WAV --> CMD
         WAV --> WH
         WAV --> HTTP
         SIDE --> BIND
         CMD --> BIND
         WH --> BIND
         HTTP --> BIND

Opcodes / CLI / makefile today

make # builds spark-stt-tts
        ./spark --dry-run examples/voice_turn.spark
        ./spark --live examples/voice_live.spark
        ./spark-stt-tts status
        ./spark-stt-tts listen --in examples/fixtures/audio/sample_review.wav
        make voice-test # local companion smoke
        

Tiny vs large

Large STT heads are owned experiments, not a claim of vendor ASR parity. See VOICE.md tiny-vs-large table.

Gaps vs roadmap

Links

VOICE.md · Spark coder (sibling train discipline) · Weight gallery · CI_PAGES.md.

Eyes — vision / image in

Plain language

Eyes would turn images into captions or visual tokens the brain can use. Spark does not have working eyes today.

Engineering stack

make test-senses
        python3 -m unittest python/sparklang/senses/test_senses.py
        

Dataflow (planned — dashed)

flowchart LR
         IMG["PNG / JPEG path"]
         ENC["Vision encoder — planned"]
         TOK["Image / caption tokens — planned"]
         EMB["spark.embed — not wired"]
         IMG -.-> ENC -.-> TOK -.-> EMB

Tiny vs large

N/A for runtime. Any future large vision experiment follows the same consumer-GPU-or-CPU discipline as coder / weights.

Gaps vs roadmap

Links

Senses package under python/sparklang/senses/ · Factory status in Factory hub · Adoption: ADOPTION_BAR.md.

Speaking — TTS / audio out

Plain language

Speaking turns text (or the last bound value) into audio the user can hear — or a dry WAV marker CI can hash.

Engineering stack

speak with model NAME loads a written voice model under out/voice_models/ (timbre/prosody params) — config / features, not a claim of neural clone training inside Spark.

spark_reply_pack can store spoken scripts for speak reply on text-only bases — still not voice-GPU / LoRA TTS.

Dataflow

flowchart TB
         TXT["Text / last_val / reply pack"]
         PCM["Local PCM synth"]
         CMD["SPARK_TTS_CMD (%o out)"]
         HTTP["HTTP TTS — SPARK_TTS_NET=1"]
         WAV["WAV path / aplay"]
         TXT --> PCM --> WAV
         TXT --> CMD --> WAV
         TXT --> HTTP --> WAV

Opcodes / CLI / makefile

./spark --dry-run examples/voice_turn.spark
        ./spark-stt-tts speak --text "hi" --out /tmp/spark-voice-test.wav
        ./spark --dry-run examples/voice_copy.spark # written model path
        ./spark --dry-run examples/model_train_reply.spark
        

Tiny vs large

Gaps vs roadmap

Production telephony gaps (barge-in, call SM, latency budgets) are documented in VOICE.md — Spark speaking is a language/demo surface, not a call-center product.

Links

VOICE.md · MODEL_TRAINING.md (reply pack) · templates under templates/voice_models/.

Thinking — LLM forward / generation

Plain language

Thinking is how Spark turns tokens into the next token / ask text — either the owned tiny CPU serve path or an optional live gateway ask (not required for dry demos).

Engineering stack — tiny CPU serve (weights)

When MLP tensors exist (Serve):

embed (mean pool) → attention (single-layer causal MHA)
         → MLP (SwiGLU) → RMSNorm → lm_head
        

Code: python/sparklang/model_lab/serve.py. HTTP/stdio: ./spark-serve-api (/v1/predict, /v1/embeddings).

Language ask / classify / extract

the AI gateway is one optional backend, not a Spark requirement. Spark is not a the AI gateway plugin.

Dataflow

flowchart LR
         IN["Tokens / prompt"]
         EMB["embed"]
         AT["attention"]
         MLP["MLP"]
         HN["RMSNorm"]
         LH["lm_head"]
         OUT["logits / text"]
         IN --> EMB --> AT --> MLP --> HN --> LH --> OUT
         GW["Optional gateway ask"] -.-> OUT

CLI / makefile

make spark-serve
        ./spark-serve docs/examples/spark-builder.sparkbc /tmp/serve-dry-001

        make spark-serve-api
        ./spark-serve-api --weights …/weights.safetensors --http \
         --host 127.0.0.1 --port 8765

        make spark-ask-http
        ./spark-ask-http --dry --model fixtures/tiny-lm --prompt "Reply: pong"
        make test-ask-gateway
        

Tiny vs large

Stub / gallery weights are not production LLMs and do measurement only. Catalog: Weight gallery.

Gaps vs roadmap

Links

Attention / forward · Architecture · Serve · Spark coder · Weight gallery · ASK_LIVE.md.

Behaviors — policies, tools, turn-taking

Plain language

Behaviors are the policies and control flow around the model — not the tensor math. This is where agentic systems win or fail: tools, SoT, abstain, expect, reply locking, train/eval feedback.

What Spark does today

Language tools (behaviors)

tool weather(city: string) -> string { "stub:local" }
        with tools [weather] {
         ask "Weather in DSM?" -> answer
        }
        

Dry ask inside an active scope returns [tool:<name>] stub:local.

Separate from factory helper tools (I/K): spark-helper-*, shadows, SDK pack — those wrap compile/decompile/serve for authors. See Diagrams and TOOLS_HELPERS.md. Model behaviors stay in .spark; helpers do not silently become policy.

Dataflow

flowchart TB
         IN["User / fixture text or transcript"]
         POL["Behavior policy
reply pack · abstain · expect"]
         ASK["ask / classify / extract"]
         TL["tool registry
with tools scope"]
         OUT["print / speak / http"]
         TR["Train STEP / model train"]
         EV["spark-eval"]
         IN --> POL --> ASK
         ASK --> TL
         TL --> ASK
         ASK --> OUT
         TR --> EV
         EV -.->|scores only; never win claim| POL

Train / eval (behavior adjacent)

Train loop · BUILD_MODELS.md · MODEL_TRAINING.md · Eval.

Gaps vs roadmap

Links

LANGUAGE.md · VOICE.md · knowledge/SAFETY_LIMITS.md (grounded / anti-guess) · ADOPTION_BAR.md · Spark coder.

Memory & context (cross-cutting)

Do not invent a context-window size for production marketing — read live dims from weights / control asm when measuring.

Comparison — Spark local SoT vs OpenBin Ask / production voice stacks

Spark is the local language + bytecode + weights SoT. Other products are related reading — not clones and not substitutes for Spark recovery / serve proofs.

Do not:

• Treat OpenBin output as verified Spark recovery
• Clone OpenBin UX into sparklang.dev
• Sell Spark voice as a phone stack

Decompile research context (OpenBin cited carefully): research/LLM_DECOMPILE.md → /docs/llm-decompile.html.

How-to lab (15 minutes)

Prove the sensory map without inventing live vendor calls:

# 0) Bootstrap
        make spark-bootstrap spark

        # 1) Ears + speaking dry (CI SoT)
        ./spark --dry-run examples/voice_turn.spark

        # 2) Companion smoke (local PCM / stub listen)
        make voice-test

        # 3) Eyes status gate (stub)
        make test-senses

        # 4) Thinking — tiny serve dry
        make spark-serve
        ./spark-serve docs/examples/spark-builder.sparkbc /tmp/serve-dry-001

        # 5) Behaviors — dry tools + expect
        ./spark --dry-run examples/tool_agent.spark

        # 6) Optional live ask — dry companion only
        make spark-ask-http
        ./spark-ask-http --dry --model fixtures/tiny-lm --prompt "pong"

        # 7) Docs regen when you edit this file
        make docs-check
        

Live STT/TTS/PSTN and live gateway ask need explicit gates — see VOICE.md and ASK_LIVE.md. Prefer passive / dry proofs when PSTN guards or missing keys apply.

Runtime / serve wiring (thinking surface)

make spark-serve-api
        ./spark-serve-api --weights …/weights.safetensors --http \
         --host 127.0.0.1 --port 8765
        

Voice companions are not inside serve_api — ears/speaking fork ./spark-stt-tts from the language VM under --live. Predict API stays token-id JSON for the tiny CPU stack.

Serve · CI_PAGES.md.

Capability status

Adoption checklist: ADOPTION_BAR.md. Roadmap sketches: ROADMAP.md.

Related docs

Quick links (Forge + Bench)

• Factory hub · Diagrams · Knowledge
• VOICE.md · AI_MODELS.md · ASK_LIVE.md
• Spark coder · Weight gallery
• Architecture · TOKENIZER.md
• LANGUAGE.md · ROADMAP.md
• Site: /docs/model-aspects.html · /docs/knowledge.html · /docs/voice.html · /docs/voice-easy.html · /docs/voice-ask.html · /docs/methods-openbin.html · /docs/spark-coder.html · /docs/weight-gallery.html