Inference — sampling, KV cache, quantization
Training writes weights. Inference turns a prompt into tokens under latency and VRAM budgets.
Prefill vs decode
- Prefill — process the full prompt; build per-layer K and V tensors (KV cache).
- Decode — for each new token, compute a fresh Q, attend to cached K/V, append new K/V, sample the next id.
Without a cache, every step would re-attend the whole prefix
(O(n^2) blow-ups). Cache memory grows with sequence length ×
layers × heads × dim — hence eviction / restore research
(e.g. RestoreKV).
Sampling
Logits → probabilities → pick a token:
| Knob | Effect |
|---|---|
| temperature | Softens / sharpens the distribution |
| top-k | Keep only k largest logits |
| top-p (nucleus) | Keep smallest set with cumulative mass ≥ p |
| greedy / beam | Deterministic or search-based (less “chatty”) |
Temperature 0 ≈ argmax. High temperature + open tools = more chaos.
Quantization
Shrink weight (and sometimes activation/KV) bit-width:
- GPTQ — post-training weight quant with Hessian-aware packing (Frantar et al.).
- AWQ — activation-aware channel protection (Lin et al.).
- NF4 / QLoRA — 4-bit NormalFloat for finetune memory (Dettmers et al.).
Accuracy vs speed is empirical — see ACL 2025 trade-off discussion (Give Me BF16…).
Spark status
Spark serve today is a tiny CPU forward plus HTTP helpers — Serve. Do not read marketing KV-cache claims into the factory. A consumer GPU is optional for experiments.
Next: Multimodal · Agents · Attention / forward.