Grader comparison: repo_scorers

Dataset 8626229b0d04e1329516c066ca53f89784c048d3400460c0ee2a039b24443173. Git commit 3b5115c3d6c2b7d73d95a631a37433aaf51563e6. inspect_ai 0.3.263.

Rates are per-case means across epochs, with 95% Wilson intervals over cases; ratios use a bootstrap over cases. Each cell is rate [low, high].

G3 (tool_call) forces the tool call with tool_choice, so its compliance measures the repo’s helper as used, not whether a model chooses to call a tool. Every grader reads verdicts case-insensitively except G3, whose helper validates the grade before upper-casing it; the case tables show how much that costs. Inspect’s OpenAI-compatible provider, which OpenRouter uses, parses a tool call written as text into a real one only under the emulate_tools model arg, which this harness does not set, so G3’s no-tool-call rate has no such rescue.

Slice: adversarial

Compliance

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	1.00 [0.98, 1.00]	0.18 [0.14, 0.24]
openai/gpt-4o-mini	2024	1.00 [0.98, 1.00]	1.00 [0.98, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.65 [0.59, 0.71]	1.00 [0.98, 1.00]
openai/gpt-4.1	2025	1.00 [0.98, 1.00]	1.00 [0.98, 1.00]
openrouter/openai/gpt-5-mini	2025	1.00 [0.98, 1.00]	0.01 [0.00, 0.03]
openrouter/openai/gpt-oss-20b	2025	1.00 [0.98, 1.00]	0.00 [0.00, 0.02]
anthropic/claude-opus-5-5	2026	1.00 [0.98, 1.00]	0.91 [0.87, 0.94]
google/gemini-3.7-flash	2026	1.00 [0.98, 1.00]	1.00 [0.98, 1.00]
openrouter/z-ai/glm-5.3	2026	1.00 [0.98, 1.00]	1.00 [0.98, 1.00]

Sample errors after retries (the failure-mode table splits them into http_4xx, http_5xx, timeout and other)

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openai/gpt-4o-mini	2024	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openai/gpt-4.1	2025	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini	2025	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b	2025	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
anthropic/claude-opus-5-5	2026	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
google/gemini-3.7-flash	2026	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3	2026	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]

Truncation

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openai/gpt-4o-mini	2024	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openai/gpt-4.1	2025	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini	2025	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b	2025	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
anthropic/claude-opus-5-5	2026	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
google/gemini-3.7-flash	2026	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3	2026	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]

Case mismatch: verdict readable ignoring case but not in the requested lower case

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	0.00 [0.00, 0.02]	n/a
openai/gpt-4o-mini	2024	0.00 [0.00, 0.02]	n/a
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.05 [0.03, 0.08]	n/a
openai/gpt-4.1	2025	0.00 [0.00, 0.02]	n/a
openrouter/openai/gpt-5-mini	2025	0.00 [0.00, 0.02]	n/a
openrouter/openai/gpt-oss-20b	2025	0.00 [0.00, 0.02]	n/a
anthropic/claude-opus-5-5	2026	0.00 [0.00, 0.02]	n/a
google/gemini-3.7-flash	2026	0.00 [0.00, 0.02]	n/a
openrouter/z-ai/glm-5.3	2026	0.00 [0.00, 0.02]	n/a

G3 failures that were case only (a Yes the helper’s schema rejected)

No replies.

Preamble: visible text before the verdict (G2, G5; replies with a verdict token)

No replies.

Q1: anchored first vs last disagreement (replies where both parse)

judge	vintage	browse_comp_2233
openrouter/openai/gpt-3.5-turbo-0613	2023	0.00 [0.00, 0.02]
openai/gpt-4o-mini	2024	0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.00 [0.00, 0.02]
openai/gpt-4.1	2025	0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini	2025	0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b	2025	0.00 [0.00, 0.02]
anthropic/claude-opus-5-5	2026	0.00 [0.00, 0.02]
google/gemini-3.7-flash	2026	0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3	2026	0.00 [0.00, 0.02]

Q1: unanchored token first vs last disagreement (replies where both parse)

No replies.

Epoch agreement on compliance / on the verdict

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	1.00 / 0.94	0.86 / 1.00
openai/gpt-4o-mini	2024	1.00 / 0.92	0.99 / 0.98
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.34 / 0.87	0.99 / 0.87
openai/gpt-4.1	2025	1.00 / 0.96	1.00 / 0.97
openrouter/openai/gpt-5-mini	2025	1.00 / 0.99	0.98 / n/a
openrouter/openai/gpt-oss-20b	2025	0.99 / 0.92	0.99 / n/a
anthropic/claude-opus-5-5	2026	1.00 / 0.96	0.85 / 0.98
google/gemini-3.7-flash	2026	1.00 / 0.99	1.00 / 0.97
openrouter/z-ai/glm-5.3	2026	1.00 / 0.99	1.00 / 0.99

At least one compliant reply in 1 / 2 / 3 attempts

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	1.00 / 1.00 / 1.00	0.18 / 0.23 / 0.25
openai/gpt-4o-mini	2024	1.00 / 1.00 / 1.00	1.00 / 1.00 / 1.00
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.65 / 0.87 / 0.96	1.00 / 1.00 / 1.00
openai/gpt-4.1	2025	1.00 / 1.00 / 1.00	1.00 / 1.00 / 1.00
openrouter/openai/gpt-5-mini	2025	1.00 / 1.00 / 1.00	0.01 / 0.01 / 0.02
openrouter/openai/gpt-oss-20b	2025	1.00 / 1.00 / 1.00	0.00 / 0.01 / 0.01
anthropic/claude-opus-5-5	2026	1.00 / 1.00 / 1.00	0.91 / 0.96 / 0.99
google/gemini-3.7-flash	2026	1.00 / 1.00 / 1.00	1.00 / 1.00 / 1.00
openrouter/z-ai/glm-5.3	2026	1.00 / 1.00 / 1.00	1.00 / 1.00 / 1.00

Cost per 100 successful grades (USD); mean output / reasoning tokens

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	$0.047; 67 / 0	$0.035; 62 / 0
openai/gpt-4o-mini	2024	$0.009; 70 / 0	$0.004; 3 / 0
openrouter/meta-llama/llama-3.1-8b-instruct	2024	$0.001; 80 / 0	$0.001; 17 / 0
openai/gpt-4.1	2025	$0.118; 64 / 0	$0.046; 3 / 0
openrouter/openai/gpt-5-mini	2025	$0.093; 423 / 327	$0.055; 247 / 129
openrouter/openai/gpt-oss-20b	2025	$0.006; 262 / 196	$0.005; 227 / 156
anthropic/claude-opus-5-5	2026	$0.524; 181 / 124	$0.406; 136 / 90
google/gemini-3.7-flash	2026	$0.095; 181 / 133	$0.073; 150 / 146
openrouter/z-ai/glm-5.3	2026	$0.425; 1390 / 1288	$0.232; 483 / 464

Failure modes (top three)

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	none	noncompliant (576)
openai/gpt-4o-mini	2024	none	noncompliant (2)
openrouter/meta-llama/llama-3.1-8b-instruct	2024	noncompliant (245)	noncompliant (3)
openai/gpt-4.1	2025	none	none
openrouter/openai/gpt-5-mini	2025	noncompliant (1)	noncompliant (701)
openrouter/openai/gpt-oss-20b	2025	noncompliant (3)	noncompliant (703)
anthropic/claude-opus-5-5	2026	none	noncompliant (61)
google/gemini-3.7-flash	2026	none	none
openrouter/z-ai/glm-5.3	2026	none	no_verdict (1)

Adversarial tag equivalent_form: agreement with the hand label

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	0.94 [0.80, 0.99]	0.02 [0.00, 0.15]
openai/gpt-4o-mini	2024	0.73 [0.56, 0.86]	0.93 [0.79, 0.98]
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.56 [0.38, 0.72]	0.84 [0.68, 0.93]
openai/gpt-4.1	2025	0.92 [0.77, 0.98]	0.97 [0.83, 0.99]
openrouter/openai/gpt-5-mini	2025	1.00 [0.89, 1.00]	0.00 [0.00, 0.11]
openrouter/openai/gpt-oss-20b	2025	0.98 [0.85, 1.00]	0.01 [0.00, 0.13]
anthropic/claude-opus-5-5	2026	1.00 [0.89, 1.00]	0.98 [0.85, 1.00]
google/gemini-3.7-flash	2026	0.98 [0.85, 1.00]	1.00 [0.89, 1.00]
openrouter/z-ai/glm-5.3	2026	1.00 [0.89, 1.00]	1.00 [0.89, 1.00]

Adversarial tag hedge: agreement with the hand label

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	0.83 [0.55, 0.95]	0.08 [0.01, 0.35]
openai/gpt-4o-mini	2024	0.97 [0.72, 1.00]	0.92 [0.65, 0.99]
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.64 [0.37, 0.84]	0.97 [0.72, 1.00]
openai/gpt-4.1	2025	1.00 [0.76, 1.00]	1.00 [0.76, 1.00]
openrouter/openai/gpt-5-mini	2025	0.97 [0.72, 1.00]	0.00 [0.00, 0.24]
openrouter/openai/gpt-oss-20b	2025	0.94 [0.68, 0.99]	0.00 [0.00, 0.24]
anthropic/claude-opus-5-5	2026	1.00 [0.76, 1.00]	1.00 [0.76, 1.00]
google/gemini-3.7-flash	2026	1.00 [0.76, 1.00]	1.00 [0.76, 1.00]
openrouter/z-ai/glm-5.3	2026	1.00 [0.76, 1.00]	0.97 [0.72, 1.00]

Adversarial tag incorrect_word_in_answer: agreement with the hand label

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	1.00 [0.76, 1.00]	0.00 [0.00, 0.24]
openai/gpt-4o-mini	2024	0.92 [0.65, 0.99]	1.00 [0.76, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.69 [0.42, 0.88]	1.00 [0.76, 1.00]
openai/gpt-4.1	2025	0.94 [0.68, 0.99]	1.00 [0.76, 1.00]
openrouter/openai/gpt-5-mini	2025	1.00 [0.76, 1.00]	0.00 [0.00, 0.24]
openrouter/openai/gpt-oss-20b	2025	1.00 [0.76, 1.00]	0.00 [0.00, 0.24]
anthropic/claude-opus-5-5	2026	1.00 [0.76, 1.00]	1.00 [0.76, 1.00]
google/gemini-3.7-flash	2026	1.00 [0.76, 1.00]	1.00 [0.76, 1.00]
openrouter/z-ai/glm-5.3	2026	1.00 [0.76, 1.00]	1.00 [0.76, 1.00]

Adversarial tag injection: agreement with the hand label

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	1.00 [0.76, 1.00]	0.72 [0.44, 0.90]
openai/gpt-4o-mini	2024	0.92 [0.65, 0.99]	1.00 [0.76, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.78 [0.49, 0.93]	1.00 [0.76, 1.00]
openai/gpt-4.1	2025	1.00 [0.76, 1.00]	1.00 [0.76, 1.00]
openrouter/openai/gpt-5-mini	2025	1.00 [0.76, 1.00]	0.00 [0.00, 0.24]
openrouter/openai/gpt-oss-20b	2025	1.00 [0.76, 1.00]	0.00 [0.00, 0.24]
anthropic/claude-opus-5-5	2026	0.86 [0.58, 0.97]	0.89 [0.61, 0.98]
google/gemini-3.7-flash	2026	1.00 [0.76, 1.00]	1.00 [0.76, 1.00]
openrouter/z-ai/glm-5.3	2026	1.00 [0.76, 1.00]	1.00 [0.76, 1.00]

Adversarial tag numeric_close: agreement with the hand label

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	0.89 [0.73, 0.96]	0.00 [0.00, 0.11]
openai/gpt-4o-mini	2024	0.58 [0.40, 0.74]	0.79 [0.61, 0.90]
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.51 [0.34, 0.68]	0.88 [0.72, 0.95]
openai/gpt-4.1	2025	0.63 [0.46, 0.78]	0.90 [0.74, 0.97]
openrouter/openai/gpt-5-mini	2025	0.91 [0.76, 0.97]	0.00 [0.00, 0.11]
openrouter/openai/gpt-oss-20b	2025	0.91 [0.76, 0.97]	0.00 [0.00, 0.11]
anthropic/claude-opus-5-5	2026	0.91 [0.76, 0.97]	0.90 [0.74, 0.97]
google/gemini-3.7-flash	2026	0.91 [0.76, 0.97]	0.89 [0.73, 0.96]
openrouter/z-ai/glm-5.3	2026	0.90 [0.74, 0.97]	0.89 [0.73, 0.96]

Adversarial tag quoted_verdict_line: agreement with the hand label

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	0.89 [0.61, 0.98]	0.00 [0.00, 0.24]
openai/gpt-4o-mini	2024	0.86 [0.58, 0.97]	0.97 [0.72, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.44 [0.21, 0.70]	0.56 [0.30, 0.79]
openai/gpt-4.1	2025	0.92 [0.65, 0.99]	0.83 [0.55, 0.95]
openrouter/openai/gpt-5-mini	2025	1.00 [0.76, 1.00]	0.00 [0.00, 0.24]
openrouter/openai/gpt-oss-20b	2025	0.94 [0.68, 0.99]	0.00 [0.00, 0.24]
anthropic/claude-opus-5-5	2026	1.00 [0.76, 1.00]	0.97 [0.72, 1.00]
google/gemini-3.7-flash	2026	1.00 [0.76, 1.00]	0.53 [0.28, 0.77]
openrouter/z-ai/glm-5.3	2026	0.97 [0.72, 1.00]	1.00 [0.76, 1.00]

Adversarial tag seed: agreement with the hand label

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	1.00 [0.89, 1.00]	0.01 [0.00, 0.13]
openai/gpt-4o-mini	2024	0.92 [0.77, 0.98]	0.99 [0.87, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.60 [0.42, 0.75]	0.98 [0.85, 1.00]
openai/gpt-4.1	2025	0.97 [0.83, 0.99]	1.00 [0.89, 1.00]
openrouter/openai/gpt-5-mini	2025	1.00 [0.89, 1.00]	0.01 [0.00, 0.13]
openrouter/openai/gpt-oss-20b	2025	0.99 [0.87, 1.00]	0.01 [0.00, 0.13]
anthropic/claude-opus-5-5	2026	1.00 [0.89, 1.00]	0.99 [0.87, 1.00]
google/gemini-3.7-flash	2026	1.00 [0.89, 1.00]	1.00 [0.89, 1.00]
openrouter/z-ai/glm-5.3	2026	1.00 [0.89, 1.00]	1.00 [0.89, 1.00]

Adversarial tag verbose_correct: agreement with the hand label

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	1.00 [0.65, 1.00]	0.43 [0.16, 0.75]
openai/gpt-4o-mini	2024	0.90 [0.54, 0.99]	1.00 [0.65, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.62 [0.28, 0.87]	1.00 [0.65, 1.00]
openai/gpt-4.1	2025	1.00 [0.65, 1.00]	1.00 [0.65, 1.00]
openrouter/openai/gpt-5-mini	2025	1.00 [0.65, 1.00]	0.00 [0.00, 0.35]
openrouter/openai/gpt-oss-20b	2025	1.00 [0.65, 1.00]	0.00 [0.00, 0.35]
anthropic/claude-opus-5-5	2026	1.00 [0.65, 1.00]	1.00 [0.65, 1.00]
google/gemini-3.7-flash	2026	1.00 [0.65, 1.00]	1.00 [0.65, 1.00]
openrouter/z-ai/glm-5.3	2026	1.00 [0.65, 1.00]	1.00 [0.65, 1.00]

Slice: simpleqa

Compliance

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	1.00 [0.98, 1.00]	0.18 [0.13, 0.24]
openai/gpt-4o-mini	2024	1.00 [0.98, 1.00]	1.00 [0.98, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.56 [0.49, 0.63]	1.00 [0.98, 1.00]
openai/gpt-4.1	2025	1.00 [0.98, 1.00]	1.00 [0.98, 1.00]
openrouter/openai/gpt-5-mini	2025	1.00 [0.98, 1.00]	0.01 [0.00, 0.04]
openrouter/openai/gpt-oss-20b	2025	0.99 [0.97, 1.00]	0.00 [0.00, 0.02]
anthropic/claude-opus-5-5	2026	1.00 [0.98, 1.00]	0.89 [0.84, 0.93]
google/gemini-3.7-flash	2026	1.00 [0.98, 1.00]	1.00 [0.98, 1.00]
openrouter/z-ai/glm-5.3	2026	1.00 [0.98, 1.00]	1.00 [0.98, 1.00]

Sample errors after retries (the failure-mode table splits them into http_4xx, http_5xx, timeout and other)

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openai/gpt-4o-mini	2024	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openai/gpt-4.1	2025	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini	2025	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b	2025	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
anthropic/claude-opus-5-5	2026	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
google/gemini-3.7-flash	2026	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3	2026	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]

Truncation

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openai/gpt-4o-mini	2024	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openai/gpt-4.1	2025	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini	2025	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b	2025	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
anthropic/claude-opus-5-5	2026	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
google/gemini-3.7-flash	2026	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3	2026	0.00 [0.00, 0.02]	0.00 [0.00, 0.02]

Case mismatch: verdict readable ignoring case but not in the requested lower case

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	0.00 [0.00, 0.02]	n/a
openai/gpt-4o-mini	2024	0.00 [0.00, 0.02]	n/a
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.05 [0.02, 0.09]	n/a
openai/gpt-4.1	2025	0.00 [0.00, 0.02]	n/a
openrouter/openai/gpt-5-mini	2025	0.00 [0.00, 0.02]	n/a
openrouter/openai/gpt-oss-20b	2025	0.00 [0.00, 0.02]	n/a
anthropic/claude-opus-5-5	2026	0.00 [0.00, 0.02]	n/a
google/gemini-3.7-flash	2026	0.00 [0.00, 0.02]	n/a
openrouter/z-ai/glm-5.3	2026	0.00 [0.00, 0.02]	n/a

G3 failures that were case only (a Yes the helper’s schema rejected)

No replies.

Preamble: visible text before the verdict (G2, G5; replies with a verdict token)

No replies.

Q1: anchored first vs last disagreement (replies where both parse)

judge	vintage	browse_comp_2233
openrouter/openai/gpt-3.5-turbo-0613	2023	0.00 [0.00, 0.02]
openai/gpt-4o-mini	2024	0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.00 [0.00, 0.02]
openai/gpt-4.1	2025	0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini	2025	0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b	2025	0.00 [0.00, 0.02]
anthropic/claude-opus-5-5	2026	0.00 [0.00, 0.02]
google/gemini-3.7-flash	2026	0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3	2026	0.00 [0.00, 0.02]

Q1: unanchored token first vs last disagreement (replies where both parse)

No replies.

Epoch agreement on compliance / on the verdict

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	1.00 / 0.97	0.80 / 1.00
openai/gpt-4o-mini	2024	1.00 / 0.97	1.00 / 0.99
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.33 / 0.89	0.99 / 0.88
openai/gpt-4.1	2025	1.00 / 0.98	1.00 / 1.00
openrouter/openai/gpt-5-mini	2025	1.00 / 0.99	0.96 / n/a
openrouter/openai/gpt-oss-20b	2025	0.98 / 0.95	1.00 / n/a
anthropic/claude-opus-5-5	2026	1.00 / 0.99	0.78 / 0.99
google/gemini-3.7-flash	2026	1.00 / 1.00	1.00 / 0.99
openrouter/z-ai/glm-5.3	2026	1.00 / 0.99	0.99 / 0.98

At least one compliant reply in 1 / 2 / 3 attempts

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	1.00 / 1.00 / 1.00	0.18 / 0.25 / 0.29
openai/gpt-4o-mini	2024	1.00 / 1.00 / 1.00	1.00 / 1.00 / 1.00
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.56 / 0.79 / 0.87	1.00 / 1.00 / 1.00
openai/gpt-4.1	2025	1.00 / 1.00 / 1.00	1.00 / 1.00 / 1.00
openrouter/openai/gpt-5-mini	2025	1.00 / 1.00 / 1.00	0.01 / 0.02 / 0.04
openrouter/openai/gpt-oss-20b	2025	0.99 / 1.00 / 1.00	0.00 / 0.00 / 0.00
anthropic/claude-opus-5-5	2026	1.00 / 1.00 / 1.00	0.89 / 0.96 / 0.98
google/gemini-3.7-flash	2026	1.00 / 1.00 / 1.00	1.00 / 1.00 / 1.00
openrouter/z-ai/glm-5.3	2026	1.00 / 1.00 / 1.00	1.00 / 1.00 / 1.00

Cost per 100 successful grades (USD); mean output / reasoning tokens

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	$0.052; 71 / 0	$0.047; 103 / 0
openai/gpt-4o-mini	2024	$0.011; 72 / 0	$0.005; 3 / 0
openrouter/meta-llama/llama-3.1-8b-instruct	2024	$0.001; 78 / 0	$0.001; 22 / 0
openai/gpt-4.1	2025	$0.136; 67 / 0	$0.070; 4 / 0
openrouter/openai/gpt-5-mini	2025	$0.082; 355 / 256	$0.060; 259 / 103
openrouter/openai/gpt-oss-20b	2025	$0.006; 231 / 157	$0.006; 217 / 110
anthropic/claude-opus-5-5	2026	$0.397; 113 / 111	$0.453; 142 / 84
google/gemini-3.7-flash	2026	$0.097; 167 / 117	$0.072; 127 / 123
openrouter/z-ai/glm-5.3	2026	$0.196; 547 / 444	$0.211; 478 / 451

Failure modes (top three)

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	none	noncompliant (491)
openai/gpt-4o-mini	2024	none	none
openrouter/meta-llama/llama-3.1-8b-instruct	2024	noncompliant (262)	noncompliant (1), no_verdict (1)
openai/gpt-4.1	2025	none	none
openrouter/openai/gpt-5-mini	2025	none	noncompliant (593)
openrouter/openai/gpt-oss-20b	2025	noncompliant (4)	noncompliant (600)
anthropic/claude-opus-5-5	2026	none	noncompliant (66)
google/gemini-3.7-flash	2026	none	none
openrouter/z-ai/glm-5.3	2026	none	noncompliant (1), no_verdict (1)

Hypothesis evidence

Verdicts on H1 and H2 are written by hand in the findings section below, citing these tables.

H1: 2026 judges, compliance by grader

judge	vintage	browse_comp_2233	sycophancy_2283
anthropic/claude-opus-5-5	2026	1.00 [0.98, 1.00]	0.89 [0.84, 0.93]
google/gemini-3.7-flash	2026	1.00 [0.98, 1.00]	1.00 [0.98, 1.00]
openrouter/z-ai/glm-5.3	2026	1.00 [0.98, 1.00]	1.00 [0.98, 1.00]

H1: reasoning judges, preamble rate on G2 and G5

No replies.

H2: compliance by judge vintage

judge	vintage	browse_comp_2233	sycophancy_2283
openrouter/openai/gpt-3.5-turbo-0613	2023	1.00 [0.98, 1.00]	0.18 [0.13, 0.24]
openai/gpt-4o-mini	2024	1.00 [0.98, 1.00]	1.00 [0.98, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct	2024	0.56 [0.49, 0.63]	1.00 [0.98, 1.00]
openai/gpt-4.1	2025	1.00 [0.98, 1.00]	1.00 [0.98, 1.00]
openrouter/openai/gpt-5-mini	2025	1.00 [0.98, 1.00]	0.01 [0.00, 0.04]
openrouter/openai/gpt-oss-20b	2025	0.99 [0.97, 1.00]	0.00 [0.00, 0.02]
anthropic/claude-opus-5-5	2026	1.00 [0.98, 1.00]	0.89 [0.84, 0.93]
google/gemini-3.7-flash	2026	1.00 [0.98, 1.00]	1.00 [0.98, 1.00]
openrouter/z-ai/glm-5.3	2026	1.00 [0.98, 1.00]	1.00 [0.98, 1.00]

Findings

To be written after reading the tables above.