Spark

Evaluation and benchmarks

Scores are instruments. Spark’s frozen harness records probes and optional frontier-API baselines as measurements. (Eval).

What benchmarks measure

Family Examples Watch-outs
Knowledge / MMLU-like Multi-task exams Contaminated train data
Coding HumanEval, MBPP Overfit to unit tests
Chat preference Arena-style Elo Style bias; judge models
Tool / agent Custom trajectories Env nondeterminism
Decompile Re-exec, recompile Pretty ≠ correct

QLoRA authors noted chatbot benchmarks can mislead (Dettmers et al.). Prefer paired comparisons on frozen prompts with disclosed judges.

Eval rules (Spark)

  1. Publish the probe set and git SHA.
  2. Separate fixture dry-run from live gateway calls.
  3. Optional frontier baseline = reference score.
  4. Abstain / expect failures are first-class — not hidden.
flowchart TB
         probes[Frozen probes] --> run[spark-eval]
         run --> scores[Numeric scores]
         run --> base[optional frontier-API baseline]
         scores --> report[Report only]
         base --> report
         report --> out[Publish scores only]

Hive home: Knowledge.