Same task, same hardware, different cell.

01 · DefinitionFive fields, no room to argue

An EvalScenario is a name, a goal string, a step cap, a wall-clock cap, and a list of required evidence substrings. That is the whole type. It is loaded straight from TOML with toml::from_str — no builder, no defaults, no optional fields.

pub struct EvalScenario {
    pub name: String,
    pub goal: String,
    pub max_steps: usize,
    pub max_seconds: u64,
    pub required_evidence: Vec<String>,
}

// Missing = a required substring no evidence item contains.
let missing_evidence = self.required_evidence.iter()
    .filter(|required| !evidence.iter().any(|item| item.contains(required.as_str())))
    .cloned().collect::<Vec<_>>();

passed: observed_steps <= self.max_steps
     && observed_seconds <= self.max_seconds
     && missing_evidence.is_empty()

The pass predicate is a three-way conjunction with no partial credit. Over budget on steps is a failure even with perfect evidence; over budget on wall clock is a failure even with a perfect step count. That bluntness is what makes cells comparable — there is no weighting to tune between runs.

The first three are verifiable tasks: their required evidence is drawn from evidence labels the harness itself produces. rust-checks is the label from plan_for_project when a Cargo.toml is found; node-fixture-test is the label for a test.js project; browser-screenshot and canvas both appear in the browser plan’s label when the page contains a <canvas>. Scoring by substring works because the labels are stable strings in src/tools/verify.rs.

The last two are different in kind, and their fixtures say so in comments. ultimate proof derivation is described as an “open-ended reasoning task (ICR sweet spot): a single hard deliverable scored by required-evidence containment over the winning artifact… No tool acceptance oracle — quality is in the artifact.” There is no command that can prove a proof is good, so the harness scores containment over the produced text and leaves quality comparison to the cross-cell judge.

02 · ExecutionTwo ways in, one way to score

eval run has two modes. Without --replay it schedules a model, builds an AgentLoop, runs the goal and records a trace. With --replay <run-id> it skips all of that and derives the same three inputs from an existing trace. Both paths hand (steps, seconds, evidence) to the identical scenario.score(…).

$ cargo run -- eval run fixtures/eval/rust.toml --cell a1_standard --out results/a1
scenario=rust verification smoke
cell=a1_standard
passed=true
steps=3
seconds=42
missing_evidence=
out=results/a1/rust-verification-smoke.a1-standard.json

$ cargo run -- eval run fixtures/eval/ultimate/proof_derivation.toml \
    --mode ultimate --cell a2_ultimate_cheap --out results/a2
scenario=ultimate proof derivation
cell=a2_ultimate_cheap
passed=true
steps=64
seconds=812
missing_evidence=
prompt_tokens=418203 completion_tokens=37116 total_tokens=455319
out=results/a2/ultimate-proof-derivation.a2-ultimate-cheap.json

$ cargo run -- eval run fixtures/eval/rust.toml --replay b1f4a9c2-7d31-4e08-9a55-0c2e6f1b83d4
scenario=rust verification smoke
passed=false
steps=5
seconds=61
missing_evidence=clippy

$ cargo run -- eval judge results/a1 results/a2 --seed 7 --out results/judge.json
pairs=5
left_wins=1
right_wins=3
ties=1
left_winrate=0.3000
out=results/judge.json

The --replay example above is the interesting one. It scores an existing trace and finds that the run took five steps against a cap of three and never produced clippy evidence. No model was contacted; the verdict came entirely from a JSONL file on disk. That is the property that makes an eval suite auditable after the fact.

03 · CellsComparable records on disk

A CellResult is what --out writes: the scenario, the cell label, the harness mode, the harness implementation, the model, the replay path, the pass/fail, the observed steps and seconds, the missing evidence, the winning artifact if there was one, and the folded upstream token counts.

Several fields carry #[serde(default)] so older result files still deserialise as the record grows. The filename is <scenario-slug>.<cell-slug>.json, slugified by collapsing any run of non-alphanumeric characters into a single dash — so a scenario called rust verification smoke and a cell called a1_standard produce rust-verification-smoke.a1-standard.json.

The A/B shape this supports is a matrix: the same scenario file, the same project, the same machine, run under two configurations and written into two directories. Everything that could differ between runs — model, mode, harness, budgets, the trace itself — is recorded in the result rather than remembered.

04 · ComparisonA blind judge that is currently a rubric

The module doc for src/eval/cell.rs is unusually candid, and the candour is the feature: “The judge here is deterministic (no live model)… A live LLM judge can replace score_pair later; the position-randomization + cross-cell pairing shape stays the same.”

So the interesting machinery is not the scoring — it is the blinding. position_coin(seed, scenario) derives a per-scenario coin flip from the seed; if it comes up swapped, the right cell is presented as slot A. The rubric only ever sees slots. The verdict is mapped back to real cells afterwards. That structure is what a live judge would slot into unchanged, and building it now means the position bias cannot be quietly introduced later.

/// Deterministic rubric over two slots (blind to which real cell each is):
/// 1. A passing result beats a failing one.
/// 2. Among equal pass/fail, fewer missing-evidence items wins.
/// 3. Then a longer artifact wins (more substance).
/// 4. Then fewer steps wins (cheaper).
/// 5. Otherwise a tie.
fn score_pair(a: &CellResult, b: &CellResult) -> Verdict {
    by_bool(a.passed, b.passed)
        .or_else(|| by_fewer(a.missing_evidence.len(), b.missing_evidence.len()))
        .or_else(|| by_more(artifact_len(a), artifact_len(b)))
        .or_else(|| by_fewer(a.steps, b.steps))
        .unwrap_or(Verdict::Tie)
}

Each comparator returns Option<Verdict> and yields None on a tie, so the chain reads as a lexicographic ordering with an explicit fallthrough. Rule three — “a longer artifact wins” — is the weakest, and it is exactly the rule a live model judge would replace. It is worth naming as a limitation rather than presenting as a quality metric: length is a proxy for substance, not a measure of it.

Reporting is small and complete. JudgeReport carries the pair count, left wins, right wins, ties, and the per-pair verdicts in scenario order. left_winrate() counts a tie as half a win and returns exactly 0.5 when there are no pairs at all — a defined value rather than a division by zero.

05 · External resultsSummarising a benchmark ledger

eval deepswe-ledger reads a JSONL ledger produced by an external benchmark lab and summarises it, classifying rows by route kind. The subcommand carries an alias (deep-swe-ledger) so both spellings work.

The classification is the point of the tool. RouteKind has four values — FairLocalNpcAgent, RouterOnly, Baseline, Other — and the summary reports fair_agent_rows and valid_fair_agent_rows separately from the totals. A ledger that mixes agent runs with router-only passthrough rows cannot be read as a single number, and this tool refuses to let it be.

$ cargo run -- eval deepswe-ledger deepswe-ledger.jsonl
rows=248
valid_rows=241
invalid_rows=7
fair_agent_rows=120
valid_fair_agent_rows=118
router_only_rows=64
route=npc-agent-local kind=FairLocalNpcAgent rows=120 valid=118 passes=43 avg_partial=0.5127 avg_f2p=0.3644 input_tokens=8412990 output_tokens=402118 steps=3914 peak_context=118204
route=router-passthrough kind=RouterOnly rows=64 valid=64 passes=0 …

$ cargo run -- eval deepswe-ledger deepswe-ledger.jsonl --json
{ "total_rows": 248, "valid_rows": 241, "invalid_rows": 7, "fair_agent_rows": 120, … }

Taken together, the eval surface is three commands and one file format. eval run produces a scored result and, optionally, a comparable record. eval judge compares two directories of those records blind. eval deepswe-ledger reads someone else’s numbers and refuses to average across incomparable route kinds. None of it depends on a model being available, which is what makes it usable as a regression suite rather than only as a benchmark.