Grader comparison harness: design

Date: 2026-10-12. Status: approved for planning, after three review rounds the same day. Scope: local only (no push, no PR).

Purpose

inspect_evals#2600 drafts a house rule for how scorers read a judge’s verdict. Draft rule 1 says to prefer a structured grader to free text. Its open question Q1 asks whether free-text parsers should take the first or the last verdict match, and the current proposal is “wherever the prompt puts it, with no default”. Matt’s comment asks for evidence across judge models of different capability and vintage, and states two hypotheses:

  • H1. 2026 models are accurate as tool-calling graders and follow explicitly requested text formats. Reasoning models may still add explanation before the verdict when it was not requested.
  • H2. Older and cheaper models follow instructions less well, and past some age do not use tool calling reliably.

This harness measures that. It runs several grader formats across several judge models on the same candidate answers and reports format compliance, mechanism failures, parser behaviour, consistency and cost, and in a second phase accuracy against labels. It is built to be re-run: new judges, new grader variants, or an eval’s real scorer can be added by configuration.

The decisions it informs: whether the lint fix in inspect-evals-lint#57 recommends the structured grader unconditionally or only for judges past some vintage; whether the free-text helper in #2598 is needed at all and whether position should have a default; and whether bounded grader retries (decision 3 in #2600) are worth mandating. It says nothing about Q2 and Q3 (upstream parity), and the summary states that.

Prior art: the HLE judge comparison post (source in Generality-Labs/Generality-Labs.github.io, blog/posts/hle-judge-comparison/). That post measured judge accuracy and cost on a hard benchmark and found every judge within a few points of the panel. This study measures the other thing: whether the judge produces a reply the scorer can read.

Two phases

Most of the decisive measurements need no ground truth. Whether a tool was called, whether the reply is valid JSON, whether the final line is correct: yes, whether there is text before a verdict-first token, whether first-match and last-match parsing disagree, and whether three epochs agree are all properties of the reply alone. Labels are needed only for accuracy and for “which parser was right when they disagreed”.

  • Phase one runs the full matrix with no labels and answers rule 1, H1, H2, the Q1 disagreement rate and the retry question.
  • Phase two labels only the cases phase one found contested and adds accuracy. It runs only if phase one shows enough disagreement to be worth pricing.

Success criteria

Phase one:

  • summary.md gives, for each grader and judge, the compliance rate, the mechanism failure rate (no tool call, schema mismatch, invalid JSON, provider rejection, truncation) and the preamble rate, with 95% intervals computed over cases.
  • It gives the first-match versus last-match disagreement rate, anchored and unanchored, for every free-text grader and judge.
  • It gives, per cell, the probability of at least one compliant reply in one, two and three attempts, from the three epochs.
  • It states for H1 and H2 whether each is supported, not supported or mixed, citing the cells that decide it.
  • A run is reproducible from its config file and the dataset’s content hash.

Phase two adds accuracy with failures counted as wrong, accuracy on parsed replies, and which parser agreed with the label on the contested cases.

First-cut matrix

Judges (all via OpenRouter; every id below is live and advertises tools, response_format and structured_outputs, checked 2026-10-12). The 2024 and 2025 rows are the grader models the repo’s evals actually hardcode most often, so the lint recommendation is tested on the judges it will be applied to. Counting quoted provider/model literals in src/inspect_evals/**/*.py on origin/main d2a1b5f85 (2026-10-12), with dated and undated ids of one model merged: the gpt-4o-mini family 22, the gpt-4o family 15, gpt-4.1 6 (against 1 for gpt-4.1-mini), gpt-5-mini 4, gpt-5-nano 3.

Model Vintage Role
openrouter/openai/gpt-3.5-turbo 2023 oldest mainstream tool-caller
openrouter/meta-llama/llama-3.1-8b-instruct 2024 small open model, tools via third-party hosts
openrouter/openai/gpt-4o-mini 2024 the repo’s most common default grader
openrouter/openai/gpt-4.1 2025 common repo grader, non-reasoning
openrouter/openai/gpt-5-mini 2025 common repo grader, reasoning
openrouter/openai/gpt-oss-20b 2025 small open reasoning model
openrouter/z-ai/glm-5.3 2026 cheap open 2026 model, the HLE post’s chosen judge
openrouter/google/gemini-3.7-flash 2026 cheap current model, HLE backup judge
openrouter/anthropic/claude-opus-5.5 2026 frontier ceiling

Graders G1 to G5, defined below, plus real repo scorers once the wrapper works. Epochs: 3. Data: about 400 cases in two slices.

Rough size: 400 cases × 5 graders × 3 epochs is 6,000 calls per judge, 54,000 in all, plus about 10,800 calls for the repo scorer rows with their longer templates. At OpenRouter’s listed prices and about 500 input tokens per call, a non-reasoning judge writes about 300 output tokens and Opus 5.5 comes to about $50. Reasoning tokens are billed as output, so at about 1,000 output tokens per call glm-5.3 is about $29, gemini-3.7-flash about $22 and gpt-5-mini about $12. gpt-4.1 at $2 in and $8 out is about $20. A realistic total is $170 to $240. Phase two labelling is three panel calls per contested case; at 200 contested cases with reasoning panel models that is about $10 to $15. The OpenRouter key has a $250 limit. The ceiling agreed with Matt is $300. The smoke run covers all nine judges and records mean input and output tokens and billed cost per judge, and the estimate is redone from that before the full run. Each judge’s share is its re-estimated cost plus 25%. Any judge that runs over its share after its first grader drops to one epoch for the rest.

Data

Slices

Slice Size Source What it measures
simpleqa ~200 SimpleQA Verified questions; answers from existing Hawk logs plus answers generated by two old or cheap models compliance and mechanism behaviour on short answers across a spread of answer quality
adversarial ~200 about 30 hand-written seeds in adversarial.yaml, expanded by templated mutation parser and judge robustness; the only slice where first and last can be expected to disagree outside bugs

HLE is not in the first cut. Forty cases gives intervals too wide to read, the Hawk HLE sets are physics-only, and HLE judging mixes format effects with knowledge gaps. The HLE post already covers judge accuracy there.

SimpleQA Verified’s three upstream grades collapse to a binary provisional label. CORRECT is yes. INCORRECT and NOT_ATTEMPTED are no. Hedged answers therefore become a test case.

Sampling is stratified by the originally recorded grade, and by answering model where a log set has several. Target mix for simpleqa: about 80 originally correct, 80 incorrect, 40 not attempted.

Answering models. Judge-hard cases come from the candidate answer, not the question. A frontier answerer gets SimpleQA Verified mostly right or cleanly abstains, which leaves the judge little to decide. An older or cheaper answerer produces near-misses, partial answers, wrong-but-confident answers, hedges and rambling, which is where judges and parsers diverge, while the reference stays easy to check. So about half the slice is taken from existing Hawk logs (current answerers) and about half is generated locally via the simpleqa_verified task in inspect_evals/simpleqa with openrouter/openai/gpt-3.5-turbo and openrouter/meta-llama/llama-3.1-8b-instruct as answerers, one epoch, provider defaults. Generated answers are graded once by the task’s own scorer so they carry an original_grade like the Hawk rows. That scorer takes its judge from get_model(role="grader") and falls back to the evaluated model when no role is bound, which would have each old answerer grade itself, so the build step passes --model-role grader=openrouter/openai/gpt-5.4, which is neither a judge under test nor on the panel. The paper’s own grader, gpt-4.1, is now a judge under test, so it is not used here. Every row records which grader produced its original_grade in source.original_grader, since the Hawk rows may have used a different one. The answering model is recorded in source.answer_model and the analysis can split on it. If Hawk lacks usable SimpleQA Verified logs, the whole slice is generated and a third, current answerer is added.

Saturation. Accuracy on SimpleQA will sit near the ceiling for every compliant judge when the answerer is strong, as the HLE post found even for small models on a harder task. That is fine for phase one, which does not measure accuracy, and the old answerers are there to keep phase two interesting. If a later run still wants harder judging, the next slice should be MATH-style answer equivalence (units, notation, ordering, algebraic form), where the judge’s task is reasoning about equivalence rather than recall, labels are partly mechanical (a CAS check), and the cases connect to the \boxed{} PRs in #2600. GPQA is a poor fit: its multiple-choice form needs no judge, and a free-form variant reintroduces the knowledge confound that kept HLE out.

Adversarial cases

Thirty or so hand-written seeds with invented questions (no benchmark content), each with a hand label. A mutation script expands the seeds, and a sample of simpleqa bases, by applying tagged transformations:

  • incorrect_word_in_answer: the candidate answer contains “incorrect” or “not correct” while being right, or the reverse.
  • quoted_verdict_line: the candidate answer contains a line correct: yes (or no) that contradicts its label.
  • injection: the candidate answer tells the grader how to mark it.
  • hedge: the answer gives two options, or declines to commit. The label becomes no.
  • numeric_close: a numeric answer just inside or just outside a reasonable tolerance (seeds only).
  • equivalent_form: the same answer in a different form (units, notation, ordering, algebraic form) (seeds only).
  • verbose_correct: a correct answer buried in long text.

A mutated case inherits its base’s label, except hedge, which sets no. The target is at least 25 cases per tag, so per-tag intervals mean something. numeric_close and equivalent_form apply to seeds only, so each numeric seed yields several variants (inside and outside tolerance, by varying amounts) and each equivalence seed several forms, to reach the target from about ten seeds each. Mutations of hand-written seeds carry provenance hand; mutations of simpleqa bases carry the base’s provisional label and are labelled in phase two if contested, like any other case.

Record format

One JSONL row per case:

{
  "id": "simpleqa-3f9c1a2b",
  "slice": "simpleqa",
  "question": "...",
  "reference": "...",
  "candidate_answer": "...",
  "source": {"log": "...", "sample_id": "...", "epoch": 1, "answer_model": "...", "original_grader": "...", "base_id": null, "mutation": null},
  "original_grade": "NOT_ATTEMPTED",
  "tags": [],
  "label": "no",
  "label_provenance": "original_grade"
}

id is a stable hash of slice, source log, sample id, epoch and mutation. label_provenance is one of original_grade (provisional, from the source log’s judge), hand, majority, reviewed.

Phase two labelling

A case is contested when, in phase one, the majority verdict of the 2026 judges across G1 to G5 disagrees with the provisional label, or the anchored first/last pair disagrees on a compliant reply from any judge. Failed replies, the substring baseline and the unanchored token pair do not count. substring disagrees with every anchored parser whenever a reasoning line contains “yes”. The token pair disagrees whenever a reply uses “no” as an ordinary word (“Yes, there is no discrepancy”), which is a parser artefact, not doubt about the case. The 2023 and 2024 judges disagree with the label often enough that counting them would contest most of the slice. Only contested cases are labelled. Expect a few percent of simpleqa plus most of the adversarial slice.

To say which token parser was right when the pair disagrees, phase two also labels a random sample of up to 50 cases with a token-pair disagreement that are not already contested, drawn with a fixed seed that is recorded in contested.json.

The panel is three frontier models that are not judges under test: openrouter/openai/gpt-6-sol, openrouter/google/gemini-3.1-pro-preview and openrouter/anthropic/claude-sonnet-5. It runs G3 (tool call) over the contested cases, one epoch, so its verdicts do not depend on a parser under test.

Votes are binary, and a panel reply that fails (no tool call, schema mismatch, error) is not a vote.

  • Three valid votes, all agreeing: that is the label (majority).
  • Two-to-one, or two-to-nothing with one failed reply, for yes: the majority is the label (majority). The HLE post found yes majorities held 44 times out of 44 under deliberation.
  • Two-to-one, or two-to-nothing with one failed reply, for no: REVIEW.md. The HLE post found the lone dissenter overturned no majorities 19 times in 57, so a no majority is not safe to take unexamined. That keeps deliberation out of the harness and puts the cases where it mattered in front of Matt instead.
  • One-to-one with one failed reply, or fewer than two valid votes: REVIEW.md.
  • Hand-labelled cases keep their label. The panel only cross-checks them. Any disagreement goes to REVIEW.md, and the label changes only if Matt changes it.

Every case sent to REVIEW.md has no label until Matt sets one (reviewed).

Matt skims REVIEW.md before the labelled dataset is frozen.

Freezing

Each dataset version is frozen with a SHA-256 content hash. Every run records the hash. The phase-one dataset is frozen before the matrix runs. Phase two writes a new version with labels filled in, and the accuracy analysis records both hashes. The dataset and REVIEW.md live in the private audit-artefacts repo under grader_comparison/datasets/<hash-prefix>/. They contain SimpleQA content and do not go into inspect_evals.

The task

grader_comparison(dataset, grader, grader_args=None, slices=None) in tools/grader_comparison/task.py.

  • Samples. input is the question. target is the reference. Metadata carries label, label_provenance, slice, tags, source and candidate_answer.
  • Solver. replay_answer() sets the candidate answer as the assistant’s output and appends it to the messages. It makes no model call.
  • Scorer. label_agreement(inner), where inner is the grader under test, resolved from the registry by grader and grader_args. The wrapper turns the inner Score into a verdict and compares it with the label:
    • CORRECT, 1, 1.0 and True give yes. INCORRECT, 0, 0.0 and False give no. NaN, None and an unscored Score give a failure, with the inner Score’s grader_failure_mode (or reason) kept.
    • A dict value is compared key by key. That is how the free-text graders report one verdict per parser.
    • The output Score’s value is a dict of 1.0 (agrees with the label) or 0.0 (disagrees or failed) per key. Its metadata holds each key’s verdict and failure mode. Metrics are mean and a failure rate per key. In phase one the label is provisional, so the agreement value is recorded but the analysis reads only the verdicts and failure modes.
  • Judge model. The graders call the task’s own model, so --model openrouter/<judge> is the judge and each log’s usage belongs to one judge. schema_tool_graded_scorer already falls back to the evaluated model when no grader role is bound.
  • Errors. The task sets fail_on_error=False. A provider rejecting a mechanism, such as an old model refusing response_format, is recorded as a sample error and reported as an outcome. It does not stop the run. The smoke run checks that a provider rejection via OpenRouter arrives as an error rather than as an output with a stop reason, and the analysis handles whichever it is.
  • Generation settings. Provider defaults for temperature, reasoning effort and max_tokens. They are recorded, not set. Old models reject large max_tokens values, so none is passed. Truncation is detected from stop_reason and reported.

Inspect reduces epoch scores before computing metrics, so the metrics in inspect view are per-sample means. The analysis reads every epoch’s Score from the log directly and does not use those metrics.

Graders

All share one criteria text. Only the output instructions differ.

Shared criteria (adapted from HLE’s judge criteria, without its extracted_final_answer framing):

Judge whether the response to the question is correct, given the correct answer. Focus only on whether the final answer in the response matches the correct answer. Do not solve the problem yourself, and do not argue for an answer other than the correct answer. Answer yes if the final answer matches the correct answer, or is within a small margin of error for a numerical problem. Answer no if there is any inconsistency, ambiguity or non-equivalence, if the response gives no final answer, or if it hedges between answers.

Registry name Mechanism Output instructions Failure handling
G1 reason_then_verdict free text “Explain your reasoning, then end your reply with a final line of exactly this form: correct: yes or correct: no.” per parser
G2 verdict_only free text “Reply with only one word: yes or no. Do not explain.” per parser
G3 tool_call schema_tool_graded_scorer, unchanged schema {explanation: str, grade: "yes" \| "no"}, explanation first its existing grader_failure_mode values
G4 structured_output GenerateConfig(response_schema=...) same schema as G3, as a JSON object grader_parse_failure, grader_schema_mismatch, grader_refusal
G5 verdict_then_reason free text “Start your reply with the single word yes or no, then give a one-sentence reason.” per parser

G5 is the format of sycophancy’s and theagentcompany’s prompts, where first-match is right by design and a reasoning model’s preamble, or a reason that restates the opposite verdict, is what breaks it. It is the only generic format where first and last can be expected to disagree on ordinary cases, so it carries most of the Q1 measurement. On a one-line G5 reply the anchored pair sees at most one line-start token and cannot disagree, so the unanchored token pair carries that evidence, and the anchored pair only disagrees when the reason starts a new line with the opposite token. G1 and G2 are kept because they are the draft rule’s own format and the simplest compliance test.

G1, G2 and G5 share one scorer, free_text_judge(instructions, label). It makes one judge call and applies every parser to the same reply. Its Score value is a dict of C, I or NaN per parser, with each parser’s failure mode in metadata. G1 passes label="correct". G2 and G5 pass no label.

G3 uses grade_map={"YES": CORRECT, "NO": INCORRECT}, since the helper upper-cases grades by default. The helper forces the call with tool_choice=ToolFunction("submit"), so G3 measures the helper as the repo uses it: the provider is told to call the tool. It does not measure whether a model chooses to call a tool unprompted, and the summary says so.

Case. Any realistic parser reads verdicts case-insensitively, so every grader’s verdict is read that way, and whether the judge used the requested case is recorded separately as a case mismatch. Every prompt and schema asks for lower case: yes and no, and the correct: label for G1. G4 lower-cases the grade before validating it, so Yes is a valid yes with a case mismatch. G3’s helper validates the payload before it upper-cases the grade, so on G3 Yes is a schema-mismatch failure. G3 keeps the helper unchanged because the helper is what is under test, and the analysis counts these case-only failures separately. That count is the evidence on whether the helper should normalise before validating.

G4 parses the reply as JSON and validates it with the same schema. Refusal is detected the way schema_tool_graded_scorer does it.

Real repo scorers. After the smoke run the matrix gains the judge prompts the #2600 PRs are arguing about, as extra rows in configs/repo_scorers.yaml. The harness branch is cut from main, so importing inspect_evals.browse_comp would load main’s code, not the PR’s. Each adapter therefore carries a copy of the PR head’s prompt template and verdict parser, pinned to the head SHA, with a provenance comment naming the PR, SHA and source path. The adapters live in repo_scorers.py. Copying the prompt and parser is enough because neither scorer needs anything else from its eval, and it keeps a run reproducible after the PR moves or merges. Neither runs as-is:

  • browse_comp, head of #2233 (59d52786c): browse_comp_scorer(scorer_model) takes a Model, so the adapter calls get_model(), which is the judge. Its value is {"score": C|I, "confidence": int}, so the wrapper takes a key argument and compares score only. The scorer defaults to no on no match, so its own Score never shows a parse failure; the raw reply is recorded and the harness’s parsers are applied to it in the analysis, which is where its failures show.
  • sycophancy, head of #2283 (b0dcbeb9e): the PR changes both of its judges through one _read_verdict. The admit_mistake judge needs a conversation with a pushback turn, which replay data does not have, so it is left out. The grade_freeform_question judge (a teacher grading a quiz, CORRECT or INCORRECT first) reads the question, the second message and the target, all of which replay provides, so the adapter wraps that function alone. That is the verdict-first shape G5 approximates, on the PR’s actual prompt and parser.

Each row is the PR’s own prompt and parser with the judge swapped, which turns a general finding into evidence on that PR.

Parsers

In tools/grader_comparison/parsers.py. Each takes the reply and an optional label, and returns yes, no or a failure. Every parser is case-insensitive. Each one that finds a verdict also reports whether the matched token, and the label for G1, was in lower case as requested.

Parser Rule Why it is here
substring yes if "yes" occurs anywhere in the lower-cased reply, else no. Never fails. the buggy pattern on main (for example agentharm/scorer.py:66), as a baseline
first_anchored the first line-start match of the verdict, case-insensitive, word-bounded. With a label, the line must read <label>: yes\|no. List markers, > quotes, heading markers and bold are allowed before it. No match fails. Q1, first
last_anchored as above, taking the last match Q1, last (the draft rule)
first_token the first word-bounded occurrence of the verdict token anywhere in the reply, no line anchor, no label. No match fails. the unanchored pattern lint#57 flags; on a one-line G5 reply the anchored pair cannot disagree, so this pair carries the Q1 evidence on G5
last_token as above, taking the last occurrence as above
strict the exact requested format. G1: the last non-empty line is correct: yes\|no (bold allowed). G2: the whole reply is the token, with an optional trailing full stop. G5: the reply starts with the token followed by punctuation or whitespace. Anything else fails. format compliance

The #2598 shared parser joins as a seventh column when it exists.

Running

run.py --config configs/<name>.yaml. A config names the judges, graders, epochs, dataset path and expected dataset hash, and the log directory. run.py checks the dataset hash, then calls eval_set over graders × judges with log_model_api=True. Inspect’s default logs the raw API request and response for only the first few calls per model, and the analysis needs every response’s usage.cost and provider. It writes run.json beside the logs with the config, dataset hash, git commit, inspect_ai version and run date.

Three configs ship:

  • configs/first_cut.yaml: the phase-one matrix above.
  • configs/repo_scorers.yaml: the real repo scorer rows, same judges.
  • configs/panel.yaml: the phase-two labelling run (G3, three panel judges, one epoch, contested cases only).

Runs are local. Hawk is the scale-up path. The same config can be rendered to a Hawk eval-set YAML later, and that is out of scope here.

Cost

Read from the logs. OpenRouter returns the billed price in usage.cost on every response (verified 2026-10-12). Inspect records the raw response in each ModelEvent but does not fold that figure into total_cost (inspect_ai#4739, deferred). The analysis sums usage.cost, adding cost_details.upstream_inference_cost for BYOK calls, as in the snippet on #4739.

The Inspect Costs Plugin prices from llm-prices, which drew its openrouter/ rows from LiteLLM until LiteLLM removed them on 2026-10-09 (BerriAI/litellm#45665). Generality-Labs/llm-prices#5 re-adds them from OpenRouter’s /api/v1/models. Once that is deployed, the plugin gives list prices and a working --cost-limit, and runs may use it as a guard. The analysis still reads billed usage.cost, because the billed price depends on which host served the call and list prices do not.

Analysis

analyse.py <log-dir> streams samples from the logs and writes:

results.csv: one row per sample × epoch × grader × judge × parser key. Columns: run_id, dataset_hash, judge, grader, parser, sample_id, epoch, slice, tags, label, label_provenance, verdict, failure_mode, agrees, sample_error, stop_reason, provider, output_tokens, reasoning_tokens, cost_usd, latency_s, case_mismatch, reply. reply is the judge’s raw text, or the tool-call arguments for G3. It is taken from the ModelEvent under the scorer span. This file is the stable contract for later comparisons.

summary.md, with slices always reported separately, in this order:

  • Compliance:
    • G1: a final correct: line is present.
    • G2: the reply is exactly the token. Preamble rate: the share of replies with text before the verdict.
    • G5: the reply starts with the token. Preamble rate as for G2. This is the H1 “explanation before the verdict” test. Preamble means visible text: OpenRouter returns reasoning separately, so a reasoning model can think at length and still comply, and H1 is about visible explanation.
    • G3: a tool was called, and the payload passed validation.
    • G4: the reply was valid JSON and passed validation.
    • All graders: sample error rate (mechanism rejected) and truncation rate.
    • All graders: case-mismatch rate, among replies whose verdict is recognisable ignoring case. For G3, also the share of failures that were case-only (Yes rejected by the schema).
  • Q1: the first versus last disagreement rate for the anchored pair and for the unanchored token pair, for G1, G2 and G5, and in phase two which one agreed with the label when they differed. The unanchored pair on G5 is the direct test of the pattern lint#57 flags.
  • Consistency and retries: the share of samples where all three epochs agree on compliance and on the verdict, and per cell the probability of at least one compliant reply in one, two and three attempts.
  • Cost: per 100 grades, plus mean output and reasoning tokens.
  • Adversarial: results per tag.
  • Hypotheses: H1 and H2, the cells that test each, and a verdict of supported, not supported or mixed.
  • Accuracy (phase two only): with failures counted as wrong, and on parsed replies only, next to the failure rate. Stated as secondary, with the saturation caveat.

All rates carry 95% intervals computed over cases, not replies. Three epochs of one case are correlated, so a Wilson interval over every reply overstates precision. Each case’s rate is averaged across its epochs first and the interval is a Wilson interval on the case count, or a bootstrap that resamples cases where the rate is a ratio of two counts.

Charts (three): a grader × judge heatmap of compliance with the mechanism failure rate overlaid; compliance against judge vintage, one line per grader; and cost against compliance per judge and grader. Phase two adds an accuracy heatmap.

Outputs go to audit-artefacts/grader_comparison/runs/<run-id>/: run.json, results.csv, summary.md and the charts. Logs stay local.

Publication

The results are meant to become a generality.org post, so the harness has to be reproducible by readers. The dataset is private because the simpleqa slice carries benchmark content, but everything else can be public: the harness code, the hand-written seeds, the mutations module, the configs, summary.md, the charts and a results.csv with the question, reference, candidate_answer and reply columns dropped for simpleqa rows. Mutations are published whole only where the base is a hand-written seed; mutations of simpleqa bases carry benchmark content and get the same column treatment as simpleqa rows. Where that public home is (a repo under Generality-Labs, not inspect_evals) is decided after the first run. The panel excludes every judge under test. Sonnet 5 shares a family with Opus 5.5, which is a weaker overlap than the HLE post’s self-grading one, and the summary states it.

Layout

tools/grader_comparison/
  __init__.py
  task.py            grader_comparison task, replay_answer, label_agreement
  graders.py         shared criteria; G1 to G5 as registered scorers; free_text_judge
  repo_scorers.py    browse_comp and sycophancy adapters, prompt and parser pinned to PR head SHAs
  parsers.py         substring, first_anchored, last_anchored, first_token, last_token, strict
  mutations.py       tagged transformations of seeds and simpleqa bases
  build_dataset.py   Hawk logs + generated answers + adversarial.yaml + mutations -> JSONL, frozen with a hash
  adversarial.yaml   hand-written seeds (no benchmark content)
  label.py           phase two: contested cases -> panel run -> labels, REVIEW.md, new frozen version
  run.py             config -> eval_set
  analyse.py         logs -> results.csv, summary.md, charts
  configs/first_cut.yaml
  configs/repo_scorers.yaml
  configs/panel.yaml
  README.md
tests/tools/grader_comparison/

tools/README.md gets an entry.

Testing

  • Parsers: a table test covering every adversarial tag, plus reason-then-verdict, verdict-then-reason with a restated verdict later in the reply, both tokens named, negated labels (incorrect: yes), numbered, bulleted, quoted and heading labels, bold, and trailing punctuation.
  • Mutations: each transformation applied to a seed produces the expected text, tag and label, and is idempotent on id.
  • Wrapper: the verdict mapping for scalar, dict, NaN and unscored inner Scores, and that failure modes are kept.
  • Replay solver: the output and messages hold the candidate answer, and no model is called.
  • End to end: one run per grader against mockllm/model with canned replies, including a G3 reply with no tool call, a G4 reply with malformed JSON, a G2 reply with a preamble and a G5 reply whose reason restates the opposite verdict. Each checks the Score values and metadata the analysis relies on.
  • Analysis: analyse.py over the end-to-end logs produces the expected rows, including cost read from a canned usage.cost, the retry probabilities and the contested-case list.

Order of work

  1. Harness and tests against mockllm.
  2. A smoke run: 5 seeds × 5 graders × all 9 judges, one epoch, 225 calls, about $1 to $2. It confirms that ModelEvents record the raw response with usage.cost and provider, how a provider rejection surfaces, and that analyse.py reads them. It also gives mean tokens and billed cost per judge for the re-estimate.
  3. Build the phase-one dataset: Hawk pull, local answer generation with the two old answerers, seeds, mutations. Freeze. No labels needed, so no review step.
  4. The phase-one run. Check spend against the estimate after the first judge completes. Add the repo scorer rows.
  5. Phase-one analysis, charts and the H1, H2 and Q1 findings.
  6. Decide on phase two from the contested count. If it runs: panel, REVIEW.md, Matt skims it, freeze the labelled version, accuracy analysis.

Out of scope

  • A Claude skill (deferred by Matt; revisit after the first run).
  • Pushing, a PR or a changelog fragment.
  • HLE or math slices. Candidates for a second run, see Saturation above.
  • Hawk runs.
  • Changing BEST_PRACTICES.md or the draft rule in #2600. The results inform that, separately.

Risks

  • Hawk data. SimpleQA Verified logs on Hawk may lack volume. Half the slice is generated locally anyway, so the fallback is to generate all of it.
  • Provider routing. OpenRouter picks a host per request, and hosts differ in tool and response_format support. The serving provider is recorded per call, so routing effects show up in the data rather than being hidden.
  • Provisional labels. Phase one’s agreement column uses the source log’s own judge as the label. It is recorded but not reported until phase two replaces it on contested cases.
  • Temperature. Provider defaults differ across judges. Three epochs measure the resulting variance, and the summary reports it.
  • Q1 outside the adversarial slice. G1 and G2 may show no first-versus-last disagreement on ordinary cases. That is itself a finding, and G5, the unanchored token parsers and the repo scorer rows are there so the question still gets pressure.