Grader parsing
October 15, 2026 · experiment · final
- Question
- How do judge models of different vintages handle free-text, tool-call and structured grader formats, and how should a scorer read their verdicts?
- Decision
- Read free-text verdicts with an anchored parser at the position the prompt asks for; no structured format worked for every 2026 judge.
- Feeds
- inspect_evals#2600, inspect_evals#2598
- Run
- 9 judges, 5 formats, 435 cases, 3 epochs
- Spend
- $74
- Detail
- spec full results rulings
How do judge models of different ages cope with different verdict formats, and how should a scorer read the verdict out of a judge’s reply?
Status: run 13-15 October 2026, about $74. This note describes the design so it can be reviewed. The main results come straight after a short overview of the method, with the detail after them. Full spec: spec.md.
Why we ran it
Many evals in inspect_evals use a model-graded scorer. A second model (the judge) reads the answer and says whether it is right, and the scorer then has to pull a verdict out of the judge’s reply. Each eval does this its own way, and some ways are buggy. For example, one scorer marks an answer correct if the word “yes” appears anywhere in the judge’s reply, including in its reasoning.
We are drafting a house rule for this (inspect_evals#2600). The draft says:
- Prefer a structured reply (a tool call or JSON) to free text.
- For free text, it is undecided whether the parser should take the first verdict it finds or the last. The current proposal is “wherever the prompt asks for it, with no default”.
There were two hypotheses:
- H1. 2026 models are reliable judges when asked to call a tool, and they follow requested text formats. Reasoning models may still write an explanation before the verdict when not asked to.
- H2. Older and cheaper models follow format instructions less well, and past some age they do not use tool calls reliably.
The decisions this feeds:
- Should our lint rule recommend structured judges for every judge model, or only for newer ones?
- Do we need a shared free-text parsing helper, and should it default to first or last match?
- Should scorers be required to retry when a judge reply cannot be read?
What it measures
Most of the questions can be answered without knowing the right answer. Did the judge reply in the requested format? Did it call the tool? Was the JSON valid? Do “first match” and “last match” give different verdicts on the same reply? Do three repeat runs agree? What does it cost? These are properties of the reply alone.
Accuracy needs ground truth, so it was measured in a second phase on a subset of cases (see “Phase two”).
How it works
Each test case is a question, a reference answer and a candidate answer that some model wrote earlier. The harness does not ask any model the question. It inserts the stored candidate answer and calls only the judge. Every judge therefore grades exactly the same answers, and differences come from the judge, the format and the parser.
In short: 9 judge models from 2023 to 2026 each graded the same 435 candidate answers 3 times, in 5 reply formats (G1 reason then a correct: yes/no line, G2 one word, G3 forced tool call, G4 JSON, G5 verdict then reason). Each free-text reply was read by 6 different parsers. A panel of three other frontier models then labelled 63 cases to measure accuracy: the 13 that phase one found contested, plus a sample of 50 where the unanchored parsers disagreed.
Results in brief
Headline: read free-text verdicts with an anchored parser (a labelled line such as correct: yes), at the position the prompt asks for. That worked for every judge, while no structured format (tool call or JSON) worked for every 2026 judge. Older judges followed formats fine; they were worse at judging.
Full write-up: first_cut/summary.md and, with accuracy, first_cut_labelled/summary.md.
Whether a parser is anchored matters much more than whether it takes the first or last match
First and last anchored match disagreed on at most 0.2% of replies. Unanchored token parsers disagreed on 17-33% of reason-then-verdict replies.
Where unanchored first and last disagreed, the one matching the prompt’s requested position was right: last on G1 (94-100% of the time), first on G5
No structured format worked for every 2026 judge
Opus 5.5 rejects forced tool calls, and GLM 5.3 wrapped its JSON in code fences on some hosts. Free text with an anchored label was at least 97% compliant for every judge.
H2 is not supported on format
gpt-3.5-turbo-0613 complied on every format. Older and smaller judges fell short on judgement instead: on the hand-labelled adversarial cases, llama-3.1-8b agreed with the label 81-86% of the time, against 96-99% for 2025 and 2026 judges.
Retries recover most format failures
Three attempts took GLM on JSON from 77% to 99% readable.
Parsers that quietly default to “no” bias weak judges
The browse_comp parser scored llama-3.1-8b’s missing correct: line as “no” on 35-44% of replies.
Design in detail
Judges (9)
| Judge | Year | Why it is included |
|---|---|---|
| gpt-3.5-turbo-0613 | 2023 | oldest mainstream model with tool calling |
| llama-3.1-8b-instruct | 2024 | small open model, served by third-party hosts |
| gpt-4o-mini | 2024 | the judge most often hardcoded in inspect_evals (22 uses) |
| gpt-4.1 | 2025 | common repo judge, no reasoning |
| gpt-5-mini | 2025 | common repo judge, reasoning |
| gpt-oss-20b | 2025 | small open reasoning model |
| GLM 5.3 | 2026 | cheap open model, the judge chosen in our earlier HLE judge study |
| Gemini 3.7 Flash | 2026 | cheap current model |
| Claude Opus 5.5 | 2026 | frontier model, as a ceiling |
The 2024 and 2025 judges were picked by counting which judge models the repo’s evals actually hardcode. Some models were called through OpenRouter, which routes each request to one of several hosts. The host was recorded on every call.
Grader formats (5)
Every format uses the same grading criteria. Only the instruction about how to reply changes. The criteria:
Judge whether the response to the question is correct, given the correct answer. Focus only on whether the final answer in the response matches the correct answer. Do not solve the problem yourself, and do not argue for an answer other than the correct answer. Answer yes if the final answer matches the correct answer, or is within a small margin of error for a numerical problem. Answer no if there is any inconsistency, ambiguity or non-equivalence, if the response gives no final answer, or if it hedges between answers.
| Id | Format | What the judge is asked to do |
|---|---|---|
| G1 | reason, then verdict | explain, then end with a line correct: yes or correct: no |
| G2 | verdict only | reply with one word, yes or no |
| G3 | tool call | call a submit tool with an explanation and a grade of yes or no |
| G4 | JSON | return a JSON object with the same two fields, using the provider’s structured-output setting |
| G5 | verdict, then reason | start with yes or no, then give a one-sentence reason |
G3 uses the repo’s existing tool-call helper unchanged. That helper forces the tool call: the provider is told the model must call the tool. So G3 tests forced tool calls, not whether a model chooses to call a tool it has merely been offered.
G5 matches the shape of some real repo prompts. It is the format where “first match” and “last match” are most likely to disagree on ordinary cases, for example when the reason restates the opposite word.
Two real repo scorers were also run, with their prompt and parser copied exactly from the open pull requests that change them and with the judge swapped: browse_comp (#2233) and sycophancy’s free-form grader (#2283). Sycophancy’s second judge was left out, because it needs a follow-up turn that replayed answers do not have.
Parsers (6)
For the three free-text formats, one judge reply is read by all six parsers, so they are compared on identical text. All are case-insensitive.
| Parser | Rule |
|---|---|
| substring | “yes” anywhere in the reply means yes, otherwise no. The buggy pattern found in the repo, kept as a baseline. |
| first anchored | the first verdict at the start of a line (bullets, quotes and bold allowed before it). For G1 the line must read correct: yes or correct: no. |
| last anchored | as above, the last match. This is the draft rule. |
| first token | the first whole-word yes or no anywhere, with no line or label requirement |
| last token | as above, the last one |
| strict | the exact requested format and nothing else. Used to measure format compliance. |
Data (435 cases)
SimpleQA Verified slice (200 cases). Short factual questions with a reference answer. About half the candidate answers come from existing logs of current models. The other half were generated for this study by gpt-3.5-turbo and llama-3.1-8b. Strong models mostly answer correctly or cleanly decline, which leaves the judge little to decide. Weaker models produce near-misses, partial answers, confident wrong answers, hedges and rambling, which is where judges and parsers start to differ. The sample was stratified to about 80 originally correct, 80 incorrect and 40 not attempted. The starting label is the original grade, collapsed to yes (correct) or no (incorrect or not attempted). A hedged answer therefore counts as no.
Adversarial slice (235 cases). 30 hand-written cases with invented questions, plus automatic variations of those and of some SimpleQA cases. Each variation has a tag, with a target of at least 25 cases per tag:
| Tag | What the candidate answer does |
|---|---|
| incorrect word | is right but contains “incorrect” or “not correct”, or the reverse |
| quoted verdict line | contains a line such as correct: yes that contradicts its true label |
| injection | tells the grader how to mark it |
| hedge | gives two options or will not commit (label becomes no) |
| numeric close | a number just inside or just outside a reasonable tolerance |
| equivalent form | the right answer in another form (units, notation, order) |
| verbose correct | a right answer buried in long text |
Repeats. Every case was graded 3 times by every judge in every format, about 58,000 judge calls, plus about 23,500 for the two repo scorers. Temperature, reasoning effort and output length were left at each provider’s default and recorded, not set.
Phase two: labels and accuracy
Only the cases phase one found “contested” were relabelled:
- the majority verdict of the 2026 judges across all formats disagreed with the starting label, or
- first and last anchored match disagreed on a well-formed reply from any judge.
A random sample of up to 50 cases where the unanchored first and last token disagreed was added, to find out which of the two was right.
These cases were labelled by a panel of three frontier models that are not judges under test: GPT-6 Sol, Gemini 3.1 Pro and Claude Sonnet 5. Each voted once using the tool-call format, so the labels do not depend on any parser under test.
- A unanimous vote, or a 2-1 vote for yes, sets the label.
- A 2-1 vote for no, or too few valid votes, goes to human review. Our earlier HLE judge study found the lone dissenter overturned a “no” majority in about a third of cases, but never overturned a “yes” majority.
- Hand-written cases keep their hand label. The panel only cross-checks them.
13 cases were contested, and with the 50-case sample the panel labelled 63. It settled 57 and could not settle 6, which were decided by hand (all six “no”). Ten labels changed in all.
Statistics
All rates have 95% intervals computed over cases, not over replies. The three repeats of one case are correlated, so treating every reply as independent would make the intervals too narrow. Each case’s rate is averaged over its repeats first, then a Wilson interval is taken over the number of cases, or a bootstrap over cases where the rate is a ratio.
What gets reported, per judge and format
- Format compliance, and the rate of mechanism failures (no tool call, wrong schema, invalid JSON, provider refused the request, reply cut off)
- Preamble rate: visible text before the verdict when none was asked for. Hidden reasoning tokens do not count.
- Whether the judge used the requested lower case
- How often first and last match disagree, for anchored and unanchored parsers
- The chance of at least one readable reply in one, two and three attempts, as a measure of what retries would buy
- Cost per 100 grades
- Results per adversarial tag
- Accuracy against the phase-two labels, counting failures as wrong and also on readable replies only
Known limits, and things worth discussing
- Only one kind of judging. Binary yes/no on short factual answers. Judging here is close to saturated, so accuracy differences are small. A harder slice (maths answer equivalence, where the judge must reason about whether two forms are the same) was proposed but not run. HLE was left out because its judging mixes format effects with knowledge gaps.
- Settings not controlled. Temperature and reasoning effort are each provider’s default, so a difference between judges mixes the model with its settings.
- Host routing. The same model served by different OpenRouter hosts can behave differently with tools and JSON. This is recorded per call, and it did show up in the results.
- Forced tool calls only. G3 does not test whether a model calls an offered tool on its own.
- Labels. Most SimpleQA labels are still the original grader’s. Only contested cases were relabelled. The panel includes Sonnet 5, from the same family as the Opus 5.5 judge.
- Adversarial cases are our own. They are hand-written and templated, not drawn from real model outputs.
- One wording. One criteria text and one wording per format. We do not know how sensitive the results are to wording.
- Few small models. llama-3.1-8b is the only small open model without reasoning.