Grader parsing
How do judge models of different vintages handle free-text, tool-call and structured grader formats, and how should a scorer read their verdicts? This experiment gathers evidence for inspect_evals#2600 (the draft parsing rule and its question Q1, first match or last), the free-text helper in #2598, and the hypotheses H1 and H2 in Matt’s comment.
Status: phases one and two done, 2026-10-13 to 2026-10-15. Spend about $74.
What was run
- 9 judges from 2023 to 2026, from gpt-3.5-turbo-0613 to Opus 5.5.
- 5 generic grader formats: reason then verdict (G1), verdict only (G2), tool call through
schema_tool_graded_scorer(G3), native structured output (G4), verdict then reason (G5). Plus two repo scorers pinned to open PRs: browse_comp #2233 and sycophancy #2283. - 6 parsers on every free-text reply: substring, first and last anchored, first and last token, strict.
- 435 cases: 200 SimpleQA Verified answers and 235 adversarial ones (30 hand-written seeds plus mutations), 3 epochs.
- Phase two: a panel of GPT-6 Sol, Gemini 3.1 Pro and Sonnet 5 labelled 63 contested cases. Matt decided the six it could not settle.
Findings
The full write-up is results/first_cut/summary.md, with accuracy in results/first_cut_labelled/summary.md.
- Anchoring decides parser quality, not position. First and last anchored matches disagreed on at most 0.2% of replies. Unanchored token parsers disagreed on 17-33% of reason-then-verdict replies.
- Position should follow the prompt. Where unanchored tokens disagreed, the last was right 94-100% of the time on G1 and the first was right on G5.
- No structured mechanism worked for every 2026 judge, on the inspect_ai of the time. Opus 5.5 rejects forced tool choice (#2695); inspect_ai has rewritten it to
autofor Claude 5 models since 0.3.270, and the runs used 0.3.263 (see the erratum in report.md). GLM 5.3 fenced its JSON on some OpenRouter hosts (#2697). Free text with an anchored label was at least 0.97 compliant for every judge. - H2 is not supported on format. gpt-3.5-turbo-0613 complied on every format. Older and smaller judges fell short on judgement instead: llama-3.1-8b agreed with the hand labels 0.81-0.86 of the time against 0.96-0.99 for 2025 and 2026 judges.
- Retries recover format failures. Three attempts took GLM G4 from 0.77 to 0.99 and Opus G2 from 0.93 to 0.99.
- Silent defaults bias weak judges. browse_comp #2233’s parser scored llama-3.1-8b’s missing
correct:line as “no” on 35-44% of replies.
The study covers binary verdicts on SimpleQA-style grading only, where judging is close to saturated.
Layout
| Path | What |
|---|---|
| spec.md | the design as agreed, 2026-10-12 |
| plan.md | the 14-task implementation plan |
| configs/ | run configs for every run in the results |
results/smoke, results/smoke2 |
smoke runs and the cost estimate |
results/first_cut |
phase one: summary, charts, review decisions, panel ids |
results/first_cut_labelled |
phase one re-analysed with phase-two labels |
results/repo_scorers |
the browse_comp and sycophancy rows |
results/datasets |
provenance of the two frozen datasets |
| results/RULINGS.md | every decision taken on Matt’s behalf during the build and runs |
| results/build-ledger.md | the task-by-task build record |
The spec, plan, provenance notes and ledger were written against the original layout in a local inspect_evals branch. Read tools/grader_comparison/ there as src/eval_experiments/grader_comparison/ here, and tools/grader_comparison/configs/ as this directory’s configs/.
Data
Not in this repository, because it carries SimpleQA Verified content. It is in the private eval-experiments R2 bucket under grader-parsing/, and MANIFEST.json lists all 755 objects (999 MB) with their sizes and SHA-256s:
| Prefix | What |
|---|---|
grader-parsing/datasets/8626229b0d04/, 3bf668d3411e/, smoke/ |
the phase-one, labelled and smoke datasets, with provenance |
grader-parsing/datasets/pools/ |
the answer pools the phase-one dataset was sampled from |
grader-parsing/logs/<run>/ |
eval logs for smoke, smoke2, first_cut, repo_scorers, panel, the answer-generation sources and the superseded partial run |
grader-parsing/results/<run>/ |
per-reply tables (results.csv, replies.csv), contested.json, REVIEW.md and the provisional analysis |
The local copies the configs point at are under ~/Developer/inspect_ai/audit-artefacts/grader_comparison/ and ~/Developer/inspect_ai/inspect_evals/logs/grader_comparison/ on Matt’s machine.
Reproduce
uv run python -m eval_experiments.grader_comparison.run --config experiments/grader-parsing/configs/first_cut.yaml
uv run python -m eval_experiments.grader_comparison.analyse <log-dir> --out <artefacts-dir>
uv run python -m eval_experiments.grader_comparison.charts <artefacts-dir>The run needs the dataset at the configured path with the configured hash, and the provider keys listed in the repository README.
To re-analyse without re-running, fetch a run’s logs from the bucket and analyse them. For example, for smoke2 (this reproduces results/smoke2/summary.md):
uv run python -m eval_experiments.store fetch --manifest experiments/grader-parsing/MANIFEST.json --dest data --prefix grader-parsing/logs/smoke2
uv run python -m eval_experiments.grader_comparison.analyse data/grader-parsing/logs/smoke2 --out data/grader-parsing/reanalysis/smoke2