Spark

Training stack — pretrain, SFT, RLHF/RLAIF, LoRA, optimizers

Industry training is a pipeline, not one fit() call. Spark’s five CPU reference methods and TinyCoder SGD are a small parallel universe — useful for reviewable .spark jobs, not InstructGPT-scale alignment theater.

Training stack schematic

1. Pretraining

Maximize next-token likelihood on huge corpora (causal language modeling). Optimizer is usually Adam / AdamW (adaptive moments) or large-batch variants; classic SGD with momentum still appears in teaching stacks and Spark fixtures.

Loss sketch: cross-entropy between predicted vocab distribution and the true next token.

2. Supervised fine-tuning (SFT)

Train on instruction → demonstration pairs so the base model follows task format. Still CE loss — just on curated data.

3. Preference alignment

RLHF — Reinforcement Learning from Human Feedback (InstructGPT / Ouyang et al., 2022; Anthropic HH): humans rank outputs → reward model → policy optimize (often PPO) with a KL penalty to the SFT policy.

RLAIF — same loop with AI preference labels / constitutions (Bai et al., Constitutional AI; Lee et al., RLAIF).

DPO and friends skip an explicit RL loop and optimize preferences directly (Rafailov et al.). Survey: LLM alignment techniques.

4. LoRA / QLoRA (PEFT)

LoRA freezes base weights and trains low-rank adapters BA (Hu et al., 2021).

QLoRA keeps the base in 4-bit (NF4) while training adapters (Dettmers et al., 2023).

Spark homepage methods are owned (not LoRA theater). See Spark coder and Train loop.

flowchart TB
         PT[Pretrain CE] --> SFT[SFT demos]
         SFT --> Pref[RLHF / RLAIF / DPO]
         Pref --> PEFT[Optional LoRA/QLoRA]
         PEFT --> Serve[Serve / eval]

Next: Inference · Evaluation.