SDD ledger — plan: /Users/matt/Developer/inspect_ai/inspect_evals/docs/superpowers/plans/2026-10-12-grader-comparison.md
Spec: /Users/matt/Developer/inspect_ai/inspect_evals/docs/superpowers/specs/2026-10-12-grader-comparison-design.md Worktree: /Users/matt/Developer/inspect_ai/inspect_evals/.worktrees/grader-comparison (branch grader-comparison, –no-track from origin/main aaa74f34a) Scope: SDD runs Tasks 1-9 (code). Tasks 10-14 (real spend, Matt checkpoints) are run by the controller inline, per Matt 2026-10-13. Task 1 Step 1 (worktree + uv sync + import check) done by controller before dispatch: inspect_evals resolves inside the worktree, inspect_ai 0.3.263.
Pre-flight scan
| Pair / task | Produces vs consumes | Finding |
|---|---|---|
| T1 -> T2,T3,T9 | records: Verdict, CaseRecord, Source, case_id, write_cases, read_cases, file_hash | consistent |
| T2 -> T4,T7 | parsers: parse_all, has_preamble, anchored, Parsed(case_mismatch), PARSER_NAMES, FreeTextFormat | consistent; T7 calls anchored(reply,“correct”,pos).case_mismatch, defined in T2 |
| T3 -> T4..T8 | task: build_task, SCORER_NAME, label_agreement metadata keys (verdicts, failure_modes, inner_metadata); helpers: make_case, mock_judge, run_task, text, scripted_reply | consistent; scripted_reply lives in helpers (T3) and is consumed in T7/T8 |
| T3 <-> T4 | T4 appends @task grader_comparison to task.py and imports graders.GRADERS | no cycle: graders imports parsers/records only |
| T4 <-> T5 | T5 extends graders.GRADERS with repo_scorers.REPO_GRADERS | no cycle: repo_scorers imports no grader module |
| T4 -> T7 | free-text metadata case_mismatch dict; G4 metadata case_mismatch bool; G3 failure_modes grader_schema_mismatch | consistent with case_columns |
| T5 -> T7 | SYCOPHANCY_LEADING, browse_comp metadata | consistent |
| T6 -> T7,T8 | run(), RunConfig, JudgeSpec (vintage, reasoning), run.json config.judges / dataset_hash | consistent with load_run |
| T7 -> T8 | load_run columns incl. compliant, verdict, anchored/token_disagree, case_mismatch, case_only_failure, vintage, reasoning | consistent with summary/_replies() test fixture |
| T9 self | mutations + build_dataset; test_build_dataset defines SEEDS_FILE itself | consistent |
| T1 self | tests vs code | consistent |
| T2 self | CASES table vs regexes | consistent on hand trace |
| T3 self | tests vs label_agreement | consistent |
| T4 self | test _run_seen vs grader code | consistent |
| T6 self | budget test readings vs loop | consistent after per-step epochs fix |
| T7 self | case_columns table vs code | consistent |
| T8 self | heading list vs summary_markdown | consistent |
| Global | mypy warn_unused_ignores=true | plan’s # type: ignore[...] comments may be unused or mis-coded -> see ruling 1 |
| Global | ruff PLR2004 outside tests | possible on == 1.0 comparisons in analyse/extract -> see ruling 2 |
Ruling: implementers adjust # type: ignore[...] comments to what mypy actually reports (add, remove or re-code them); mypy must be clean — warn_unused_ignores=true makes plan-supplied ignores possibly wrong — cost if wrong: none beyond comment churn. Ruling: if ruff PLR2004 flags a float literal comparison in non-test code, hoist it to a named module constant rather than add a noqa — repo convention is no magic comparisons outside tests — cost if wrong: trivial. Ruling: the VIRTUAL_ENV mismatch warning uv prints in the worktree is harmless (uv uses the worktree .venv); implementers ignore it — cost if wrong: none, import path verified.
Progress
Task 1: implementer ae968796e94b5efbd (haiku); review ae0b09fa0410609cb (sonnet): spec ✅, quality Approved; ⚠️ trailer confirmed as a real gap (Haiku 4.5 instead of mandated Opus 5.5). Task 1: minor (deferred): file_hash/read_cases lack one-line docstrings while siblings have them. Task 1: minor (deferred): no test for write_cases([]) or read_cases on a missing/malformed file. Task 1: fix round 1/5 (1 addressed, 0 open — commit trailer; amend 9c2b9dfd9 -> ed26e7374, tree diff empty) Ruling: no reviewer re-dispatch for Task 1’s fix round — the fix only rewrote the commit message; controller verified git diff 9c2b9dfd9 ed26e7374 is empty and the trailer reads Opus 5.5 — cost if wrong: none, the tree is byte-identical to the reviewed one. Ruling: every later implementer dispatch states the exact trailer text explicitly, because the haiku implementer wrote its own model name — cost if wrong: none. Task 1: complete (commits aaa74f34a..ed26e7374, review clean) Task 2: implementer a515b71480063d705 (haiku); review a61754ac3b788a8fc (sonnet): spec ✅, quality Approved; ⚠️ trailer — controller verified Opus 5.5 trailer on 186d220ea before review. Task 2: minor (deferred): anchored() docstring says “after <label>” (brief has <label>:). Task 2: minor (deferred): _LINE_PREFIX list-marker * and emphasis * alternations overlap; harmless on all traced cases, worth a comment. Task 2: complete (commits ed26e7374..186d220ea, review clean) Task 3: implementer a4b9cc2d780b0d856 (sonnet) DONE 18985f540; controller checked: trailer Opus 5.5; suite’s 1 warning is PydanticDeprecatedSince20 from a third-party import in the inspect_evals import chain, not task code. Task 3: review ade2a8feabe8b043d (sonnet): spec ✅, quality Approved. Task 3: minor (deferred): _failure_modes uses or, so an empty-string grader_failure_mode falls through to reason. Task 3: minor (deferred): compare_keys with a scalar inner value yields all-None items silently (no caller does this). Task 3: minor (deferred): metadata inner_value stores the unfiltered value when compare_keys narrows the comparison; worth a comment. Task 3: minor (deferred): no multi-sample test of per_key_mean or standalone test of json_safe. Task 3: complete (commits 186d220ea..18985f540, review clean) Task 4: implementer a3673b0671431690e (sonnet) DONE e092040fa; review a878e278161e6711a (sonnet): spec ✅, quality Approved; trailer Opus 5.5. Inspect 0.3.263 records a scorer exception as a per-sample error with log.status “success” under fail_on_error=False (test passed as written). Task 4: minor (deferred): graders.py imports private _is_grader_refusal from inspect_evals.utils.scorers (plan-mandated); consider a public name. Task 4: complete (commits 18985f540..e092040fa, review clean) Task 5: implementer a23469c9205161b80 (sonnet) DONE 5d357fe77; review a0598a5e3dce5b3d9 (sonnet): spec ✅, quality Approved; reviewer re-fetched pinned sources, byte-for-byte match; trailer Opus 5.5. Task 5: ⚠️ browse_comp config row compare_keys [score] — resolved by controller: plan Task 12 Step 1 writes it in configs/repo_scorers.yaml (Task 12 is controller-run). Task 5: minor (deferred): _read_verdict copied without its source docstring; pull it in verbatim. Task 5: complete (commits e092040fa..5d357fe77, review clean) Task 6: implementer a545d297d0516a4e4 (sonnet) DONE 92a64c214; review a90fb4886816311b2 (sonnet): spec ❌ / Needs fixes — 2 Important, both plan-mandated. Ruling: run.json steps stays an append-only invocation log (each re-run appends, latest record per (judge, grader) is current; previous dict already takes the latest) — re-run spend is worth recording; fix is documentation plus an explicit test assertion on the history, not de-duplication — cost if wrong: a reader of run.json must take the last record per pair. Ruling: the budget cut must survive a crash between a judge’s first and second grader — persist cut_after: true on the triggering step and restore epochs=1 for that judge on resume; add a crash-window test — spec says a judge’s graders not yet run keep the reduced count — cost if wrong: one extra run at full epochs for one judge. Task 6: minor (deferred): no multi-judge test; _git_commit depends on cwd; check_dataset raises raw FileNotFoundError for a missing dataset. Task 6: fix round 1/5 (2 addressed, 0 open — append-only steps documented + full-history test; cut_after persisted + interruption test; commits 92a64c214..69e82fe0a; re-review a43e9f5834fab5c13) Task 6: complete (commits 5d357fe77..69e82fe0a, review clean) Task 7: implementer acb7e67dbbf4920d0 (sonnet) DONE a00156374; review a5e454caf4fe13c8c (sonnet): spec ✅, quality Approved; scorer span type “scorer”/“scorers” confirmed on 0.3.263; trailer Opus 5.5. Task 7: minor (deferred): read_log relies on label_agreement always populating verdicts; an empty dict would silently emit zero rows. Worth a comment. Task 7: complete (commits 69e82fe0a..a00156374, review clean) Task 8: implementer a8816b75021e290b2 (sonnet) DONE 6120da70a; review ae5ccffac3a1c5532 (opus): spec ✅ vs brief, quality Needs fixes — 2 Important (plan-mandated), 3 ⚠️. Ruling: epoch verdict agreement counts only compliant replies, over cases with >=2 compliant epochs — an all-failed judge must not score 1.00 consistency — cost if wrong: none, the old metric was misleading. Ruling: “which parser was right” uses only non-provisional labels (label_provenance != original_grade) and per-case rates with intervals via case_rate — spec says intervals over cases and this table comes from phase-two labels — cost if wrong: smaller n in that table. Ruling: browse_comp_2233 joins the anchored Q1 table (not the token table) but does not contest cases — Task 13’s findings need #2233’s anchored disagreement rate; the contested trigger is defined over G1-G5 — cost if wrong: repo-row disagreements unlabelled in phase two. Ruling: labelled-mode accuracy keeps all labelled cases (uncontested keep provisional labels by design) and the accuracy section states the count per label_provenance — cost if wrong: a reader may over-trust provisional-heavy cells; the counts make it visible. Ruling: the token sample draws from all non-errored replies (compliant or not) — unanchored parsers matter most on replies that ignore the format — cost if wrong: a few more phase-two labels. Ruling: fix minors 5 (nan reasoning tokens in the cost cell), 6 (stray empty adversarial tag section) and 7 (one 2026 constant) in this round — trivial output bugs — cost if wrong: none. Task 8: minor (deferred): no intervals on at-least-one and epoch-agreement cells. Task 8: minor (deferred): at_least_one can mix case sets across k when epochs are missing (latent). Task 8: minor (deferred): test gaps — bootstrap bounds weakly asserted, case_rate interval unchecked, contested edge cases (failed reply with anchored disagreement, non-GENERIC grader, token cap, seed in output), summary values unasserted. Task 8: fix round 1/5 (6 addressed, 0 open — verdict agreement over compliant epochs; parser-right table per case on non-provisional labels; browse_comp in anchored Q1; provenance counts; token sample from all non-errored replies; output fixes; commits 6120da70a..7b37b8bf0; re-review a32d011f70d90d141) Task 8: minor (deferred): agreement cells render “nan” instead of “n/a” when no eligible cases. Task 8: complete (commits a00156374..7b37b8bf0, review clean) Task 9: implementer ac66c0e701ae0d542 (sonnet) DONE 8b98041ca; review a686763a9cdf650ab (opus): ❌ / Needs fixes — 2 Important, 2 ⚠️; all 21 new seed labels checked and defensible. Ruling: rename build_dataset.py -> make_dataset.py and test_build_dataset.py -> test_make_dataset.py, and narrow model_roles[“grader”] (ModelConfig | list[ModelConfig]) — the repo mypy exclude regex “build” silently skips any path containing “build”, hiding a real union-attr error; controller updates the plan’s Task 10/11 commands — cost if wrong: one rename. Ruling: replace the four integer-reference numeric seeds (0 °C, 365 days, 50 m, 64 squares) with decimal-valued quantities, each with rounding-level yes and near-miss no variants — numeric_close exists to test the tolerance boundary — cost if wrong: none. Ruling: fold in the cheap minors this round: swap plain-longest-river for an undisputed fact; replace “The British dollar” distractor; one cheetah variant as the bare synonym “Acinonyx jubatus.”; mutate_bases prints a shortfall like sample_simpleqa — cost if wrong: none. Ruling: mutants of a simpleqa base whose label phase two corrects must follow the base (hedge stays no; verbose_correct dropped if the base becomes no) — carried to Task 14 (label.py, controller-run) — cost if wrong: stale labels on a few mutants in accuracy tables. Task 9: minor (deferred): equivalence yes-variants at H2O and 12 still repeat the reference token. Task 9: fix round 1/5 (3 addressed, 1 open — numeric-inch-to-centimetres has no rounding-level yes variants (controller’s own example values caused it); commits 8b98041ca..cc91c51e6; re-review a7d5a5d434e60c88b) Task 9: minor (deferred): _grader_model list branch untested. Task 9: out-of-scope (deferred): repo mypy exclude “build” is an unanchored regex skipping any path containing “build”; anchor as (?:^|/)build/ in a separate inspect_evals change. Task 9: fix round 2/5 (1 addressed, 0 open — pi seed replaces inch seed; near-miss no’s tightened; commits cc91c51e6..652115194; re-review ada395f7bf7044982) Task 9: complete (commits 7b37b8bf0..652115194, review clean) Final review a9a248b3c3f7a8091 (fable): With fixes — 1 Critical, 5 Important, 9 Minor. Ruling: fix Critical 1 (–judges rewrites run.json judges) by filtering in run() without mutating config, merging judge lists in _load_info, plus a multi-judge multi-invocation test — it would corrupt the contested list and H1/H2 tables in Task 12’s per-judge workflow — cost if wrong: none. Ruling: loosen strict to accept emphasis around the label and token and an optional trailing “.” or “!” for G1/G2/G5, but not list markers — decoration does not impair a scorer’s reading and the spec allows bold; a list marker before G1’s final line remains a format deviation — cost if wrong: compliance slightly higher than a byte-exact reading; the anchored parsers still show the raw spread. Ruling: SYCOPHANCY_LEADING mirrors the PR’s leading regex (optional short label before the token, same token group, no flags) — compliance must mean “the PR’s leading branch decided” — cost if wrong: none. Ruling: export browse_comp’s verdict_found as a replies column pr_verdict_found and use it as browse_comp’s compliance; harness anchored pair stays for the Q1 table — Task 13 needs the silent-default rate — cost if wrong: none. Ruling: accuracy uses compliant for parsed/correct (fixes browse_comp’s default “no” counting as parsed) — cost if wrong: none (phase two only). Ruling: pass retry_on_error=2 to eval_set and add an error_kind replies column (http_4xx / http_5xx / timeout / other from the error message), shown as sample_error:correct: yes**); harmless loosening. Final: minor (deferred): a caller-supplied retry_on_error in eval_set_args would raise a duplicate-keyword TypeError (same as retry_attempts). Ruling: keep this plan’s SDD workspace and ledger until Tasks 10-14 finish (they continue the same plan, controller-run); skip finishing-a-development-branch for now because Matt asked to keep the branch local with no merge or PR — cost if wrong: none. Tasks 1-9: complete (commits aaa74f34a..fd13b4254, final review clean after one fix wave). Ruling: real runs use uv run --with openai (Inspect’s OpenRouter provider needs the openai package, absent from the dev group); README documents it; pyproject untouched to keep the change local — cost if wrong: none. Smoke attempt 1 failed before any call (PrerequisiteError), $0 spent. Task 10 smoke attempt 2: every gpt-3.5-turbo call failed 404 — OpenRouter workspace guardrail (provider-not-allowed + ZDR) excludes OpenAI’s own endpoint, the only host for openai/gpt-3.5-turbo. Probed all judges, panel and gpt-5.4: only gpt-3.5-turbo (and gpt-4-turbo) blocked. Llama logs confirm provider, billed usage.cost and replies land in the logs. Ruling: swap the 2023 judge to openrouter/openai/gpt-3.5-turbo-0613 (June 2023 snapshot, introduced function calling; served via Azure, passes the guardrail) instead of asking for a guardrail change — same vintage and role, no account change — cost if wrong: Matt may prefer the guardrail change; trivially reversible. Finding (for the Task 10 checkpoint): run.py’s spent_usd reads 0.0 for every step; the key usage endpoint lags or rounds small amounts, so the budget cut may never trigger. Proposed fix for Matt: compute step spend from the step’s logs (sum of billed usage.cost) instead of the key endpoint. Finding (for the Task 10 checkpoint): a guardrail-blocked judge makes every sample a sample_error while eval_set reports success; proposed guard: stop a judge after its first grader if every sample errored, printing the first error. Task 10 smoke attempt 3: 45/45 steps, 225 replies, $0.094 billed; provider and usage.cost present on every non-error reply. Projected phase one $39.75 (estimate.md in audit-artefacts runs/smoke). Configs committed 928da3364. Task 10 checkpoint findings for Matt: (a) Opus 5.5 G3 = 100% sample error, host rejects forced tool_choice; (b) gpt-4o-mini G3 = 100% 404, guardrail leaves no tool-capable endpoint; (c) Opus and Gemini G4 return ```json-fenced JSON despite strict response_format (host not enforcing); (d) gpt-oss-20b G3 replies have finish_reason “error” mid-generation and are scored as grader_no_tool_call; (e) spent_usd stays 0 (key usage endpoint), so the budget cut is inert; (f) no guard for an all-errored step. Task 10: paused at checkpoint, awaiting Matt. Task 10a: implementer abbbaff1dd35fde9e (sonnet) DONE 87f78a25b; review a68e7422f67da2a8c (sonnet): spec ✅, Approved. Task 10a: minor (deferred): call_cost docstring could state billed-wins precedence. Task 10a: complete (commits 928da3364..87f78a25b, review clean) Ruling (Matt supplied a new OpenAI key; llm-prices now prices openrouter/ ids): judges gpt-4o-mini, gpt-4.1 -> openai/ direct; Opus 5.5 -> anthropic/claude-opus-5-5 direct; gpt-3.5-turbo-0613, llama-3.1-8b, gpt-5-mini (needs OpenAI org verification direct), gpt-oss-20b, glm-5.3, gemini-3.7-flash stay on OpenRouter with provider.require_parameters=true; runs add –with inspect_costs_plugin and INSPECT_COSTS_API_URL=https://llm-prices.generality.org/api/inspect-costs — cost if wrong: one more smoke rerun (~$0.10). Ruling: gpt-oss-20b on OpenRouter ignores the CoreWeave host (provider.ignore [CoreWeave]) — CoreWeave returns finish_reason error with an empty message on every forced tool call, while Bedrock, DeepInfra and Groq return proper tool calls (probed 2026-10-13); a host fault, not the model — cost if wrong: none; recorded as a finding. Ruling: Gemini 3.7 Flash -> google/gemini-3.7-flash direct — via OpenRouter its strict response_format was not enforced (fenced JSON every time, with or without require_parameters) while Google’s API enforces the same schema (probed 2026-10-13); Opus 5.5 G3 stays a 100% sample error because Anthropic’s own API rejects forced tool_choice for the model (a real finding) — cost if wrong: one more Gemini smoke rerun. Task 10: smoke2 complete — 225 replies, 9 judges; compliance 1.0 on every cell except Opus 5.5 G3 (Anthropic rejects forced tool_choice: real finding). Cost present on every reply; projected phase one $39.31 (estimate.md, runs/smoke2). Superseded OpenRouter-Gemini logs moved to logs/grader_comparison/superseded/. Configs committed. Task 10: complete; Task 11 awaits Matt’s go-ahead. Plan updated (controller): Tasks 11+ commands carry –with openai / anthropic>=1.0.0 / google-genai / inspect_costs_plugin; judge_budgets dropped from Task 12 (key usage endpoint inert; spend checked from log costs); provisional grader openai/gpt-5.4 direct. Task 11a: implementer accf7a7679fee1769 (sonnet) DONE 6330a2d74 — imported Epoch logs record the grader as scorer param grader_model (google/gemini-2.0-flash-001), not model_roles; review a8e3744c9ef2e0a36: Approved, real log resolves correctly. Task 11a: complete (commits b2ab7dab1..6330a2d74, review clean) Task 11: Hawk set imported-sqav-epoch-compa-ethmbrd1xwsda861 downloaded (48 evals, ~20 current answering models, 100 MB) to logs/grader_comparison/sources/hawk/. Ruling: generated answers use openrouter/openai/gpt-3.5-turbo-0613 (plain gpt-3.5-turbo is guardrail-blocked on OpenRouter) and openrouter/meta-llama/llama-3.1-8b-instruct, graded by openai/gpt-5.4 direct — same answerer roles as the plan — cost if wrong: one regeneration (~$5). Task 11b: implementer a93841f62fa119fa7 (sonnet) DONE 1763ca331; review a252b20c6f67f03ea (sonnet): Needs fixes — 1 Important (share or default treats :0 as unset). Ruling: a pool that cannot fill its share stays short (no redistribution) but the warning names the pool, with a shortfall test — the real pools fill their shares (generated CORRECT 51 >= 40) and silent redistribution would blur the half/half design — cost if wrong: a slightly short slice, visible in the printout. Ruling: reject an empty –exclude-grader string (it would match every row) — cost if wrong: none. Task 11b: minor (deferred): _split_targets can go negative with many tiny pools; round() is banker’s rounding; no CLI parsing tests. Task 11b: fix round 1/5 (3 addressed — :0 share kept, pool-named shortfall + test, empty exclude rejected; commits 1763ca331..3b5115c3d; re-review a052c4953cc9f54e3). Task 11b: complete (commits 6330a2d74..3b5115c3d, review clean) Task 11: complete — phase-one dataset frozen at audit-artefacts/grader_comparison/datasets/8626229b0d04/dataset.jsonl (sha256 8626229b0d04e1329516c066ca53f89784c048d3400460c0ee2a039b24443173): 200 simpleqa (100 Hawk across 28 models, 100 generated) + 235 adversarial; PROVENANCE.md beside it. audit-artefacts is not a git repo, so the plan’s commit step does not apply. Task 12: llama-3.1-8b G4 throttled — only CoreWeave offers strict structured_outputs for llama on OpenRouter, so require_parameters funnels all G4 traffic there; 35/1305 samples in 40 min (429 backoff). Ruling: stop batch 1, run the other judges first, then retry llama’s G4/G5 at max_connections 4 (separate config, same log_dir); if still throttled, run llama G4 at 1 epoch — keeps require_parameters so the measurement stays a real mechanism result — cost if wrong: llama G4 has fewer epochs than other cells. Task 12: batches 1-3 done, $41.06 billed across 8 judges (GLM 5.3 $10.85 vs $5.18 projected: more reasoning tokens on real answers). Opus G3 1305 sample errors as expected. Task 12: llama G4 at 3 epochs ran 13 samples/10 min (CoreWeave 429s); per ruling, G4 runs at 1 epoch, max_connections 4 (configs/first_cut_llama_g4.yaml); G5 at 3 epochs, 20 connections (configs/first_cut_llama_g5.yaml). Partial 3-epoch G4 log moved to logs/grader_comparison/superseded/. Task 13 (charts): implementer ae01939b074a19634 (sonnet) DONE_WITH_CONCERNS 48cfd504e; review aecb35db623198faf: spec ✅, Approved (dataviz rulings met; label spreading justified). Task 13: minor (deferred): no tests for _spread / >8-grader ValueError / slot kept when a grader is absent; style passed via DataFrame.attrs; LABEL_ROOM_X hard-coded; empty simpleqa slice raises IndexError; heatmap ink/surface switch at 0.6 gives ~3.6:1 just below; module-level rcParams mutation. Task 13a: implementer afd6d4533d78fd8ad (sonnet) in progress — merge graders in run.json, summary uses graders present in data. Task 13: fix round 1/5 (mypy without matplotlib: import ignores with unused-ignore; efe3f58ef). Ruling: controller verification replaces the scoped re-review for this comment-only fix — ran mypy with and without matplotlib on all 27 files (clean), ruff (clean), full suite with matplotlib (206 passed) — cost if wrong: none. Task 13 (charts code): complete (commits 3b5115c3d..efe3f58ef, review clean) Task 13a: review add4c9c3f6fa20e85: spec ✅, Approved. Task 13a: minor (deferred): charts._simpleqa takes graders from run.json only (now merged, so covered); summary test checks headers only. Task 13a: complete (1b40d11d5, review clean) Task 12 provisional findings (first_cut, all judges except llama G4; runs/first_cut_provisional): free-text compliance ~1.00 for every judge incl. gpt-3.5-0613 (H2 not supported on format following); Opus G2 0.93 (7.3% visible preamble, last-line verdict still parseable), Opus G3 0.00 (forced tool_choice rejected); GLM G4 0.77 (fenced JSON despite require_parameters); llama G1 0.97 (no final correct: line). Anchored first/last disagreement ~0 everywhere; unanchored token first/last disagreement 17-33% on G1 for gpt-3.5/4o-mini/4.1/GLM (lint#57 evidence). G5 case mismatch: Gemini 0.998, GLM 0.945, llama 0.47, 4o-mini 0.34, 4.1 0.35, Opus 0.21 — case-sensitive parsing would fail them. Ruling: restored logs/grader_comparison/first_cut/run.json config.graders to the five phase-one graders (backup in logs/grader_comparison/superseded/) — llama G4 ran on pre-13a code and overwrote the list; steps untouched — cost if wrong: none, the data is unchanged. Task 13: fix round 2 (cost ticks in significant figures) verified by controller: suite with matplotlib, mypy clean; charts regenerated with PNG twins. Findings written into runs/first_cut/summary.md. Task 14: panel routing ruling — gpt-6-sol (OpenAI direct; Inspect uses the Responses API, which allows forced tool calls), gemini-3.1-pro-preview (Google direct), claude-sonnet-5 (Anthropic direct); all three accept forced tool_choice (probed 2026-10-15), priced by llm-prices — cost if wrong: none. Panel runs G3 over 63 cases (13 contested + 50 token sample) at 1 epoch. Task 14: implementer a4ef6cd8c4492af17 (sonnet) DONE 99eacb0d2 (trailer wrong: Sonnet 5.5); review ac973c40df8ab3956 (opus): Needs fixes — 1 Critical, 3 Important. Ruling: normalise YAML decision values (True->yes, False->no; only yes/no accepted) and re-validate every updated record — REVIEW.md’s own instruction produced booleans — cost if wrong: none. Ruling: a contested mutant’s own panel outcome wins over propagation from its base; when they disagree or the mutant is unresolved, add a review entry naming the conflict — mutant panel votes are evidence about that case — cost if wrong: a few more review entries. Ruling: drop review entries for mutants removed from the output (quoted_verdict_line / verbose_correct on a flip) — cost if wrong: none. Ruling: pad each case’s votes to the panel size from run.json judges with None — a missing judge must never make a unanimous no — cost if wrong: none. Ruling: Matt’s explicit decision may override a hand label (he is the authority), but REVIEW.md states hand entries are informational unless he intends an override; warn on decision ids that are unknown or already settled; REVIEW.md lists a base’s mutants and a mutant’s base — cost if wrong: none. Panel run: 189 replies, 0 errors, $0.42 (gpt-6-sol yes 41/no 22, gemini-3.1-pro 44/19, sonnet-5 47/16). Task 14: trailer amended (99eacb0d2 -> 0b2aee991, tree unchanged); fix round 1/5 (7 addressed — YAML normalisation + revalidation, mutant own outcome wins with conflict entries, dropped mutants filtered, votes padded, hand entries informational, decision warnings, base/mutant links; commits 0b2aee991..db260f178; re-review a4b93d3254af202fe). Task 14 code: complete (commits 95e9ff071..db260f178, review clean) Task 14: minor (deferred): review reason for a decided pre-contested mutant keeps its first-pass reason; spurious conflict flag on hedge mutants when the base flips; conflict entries don’t block the CLI (by ruling); YAML on reads as yes; a case with no replies keeps its label silently. Task 14: phase-two dataset frozen at datasets/3bf668d3411e (sha256 3bf668d3411ea676270dd5dcac1dbd19e27669be8dc71ff1b3dcbbd6b15fa8f1); Matt decided all 6 review cases no (2026-10-15). Task 14: complete — accuracy analysis in runs/first_cut_labelled; findings in runs/first_cut/summary.md updated with phase two (Q1: last token right on G1 94-100%, first token right on G5; Sonnet 5 alone accepted hedges). Task 14: minor (deferred): fmt_rate can print ‘-0.00’ for a lower bound.