Grader comparison: first_cut

Dataset 8626229b0d04e1329516c066ca53f89784c048d3400460c0ee2a039b24443173. Git commit 3b5115c3d6c2b7d73d95a631a37433aaf51563e6. inspect_ai 0.3.263.

Rates are per-case means across epochs, with 95% Wilson intervals over cases; ratios use a bootstrap over cases. Each cell is rate [low, high].

G3 (tool_call) forces the tool call with tool_choice, so its compliance measures the repo’s helper as used, not whether a model chooses to call a tool. Every grader reads verdicts case-insensitively except G3, whose helper validates the grade before upper-casing it; the case tables show how much that costs. Inspect’s OpenAI-compatible provider, which OpenRouter uses, parses a tool call written as text into a real one only under the emulate_tools model arg, which this harness does not set, so G3’s no-tool-call rate has no such rescue.

Slice: adversarial

Compliance

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openai/gpt-4o-mini 2024 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.96 [0.93, 0.98] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openai/gpt-4.1 2025 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/openai/gpt-5-mini 2025 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/openai/gpt-oss-20b 2025 0.99 [0.97, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.98, 1.00] 0.97 [0.94, 0.99] 0.00 [0.00, 0.02] 1.00 [0.98, 1.00] 0.99 [0.97, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 0.82 [0.77, 0.86] 1.00 [0.98, 1.00]

Sample errors after retries (the failure-mode table splits them into http_4xx, http_5xx, timeout and other)

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4.1 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 1.00 [0.98, 1.00] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

Truncation

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4.1 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

Case mismatch: verdict readable ignoring case but not in the requested lower case

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.02 [0.01, 0.04]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.39 [0.33, 0.46]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.02] 0.02 [0.01, 0.04] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.59 [0.53, 0.65]
openai/gpt-4.1 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.22 [0.17, 0.28]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.02 [0.01, 0.04] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.02] 0.02 [0.01, 0.05] n/a 0.00 [0.00, 0.02] 0.11 [0.08, 0.16]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.01 [0.00, 0.03] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.99 [0.97, 1.00]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.91 [0.87, 0.94]

G3 failures that were case only (a Yes the helper’s schema rejected)

judge vintage tool_call
openrouter/openai/gpt-3.5-turbo-0613 2023 n/a
openai/gpt-4o-mini 2024 n/a
openrouter/meta-llama/llama-3.1-8b-instruct 2024 n/a
openai/gpt-4.1 2025 n/a
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.79]
openrouter/openai/gpt-oss-20b 2025 n/a
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 n/a
openrouter/z-ai/glm-5.3 2026 n/a

Preamble: visible text before the verdict (G2, G5; replies with a verdict token)

judge vintage verdict_only verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4.1 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.03 [0.01, 0.06] 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

Q1: anchored first vs last disagreement (replies where both parse)

judge vintage reason_then_verdict verdict_only verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4.1 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

Q1: unanchored token first vs last disagreement (replies where both parse)

judge vintage reason_then_verdict verdict_only verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.28 [0.23, 0.34] 0.00 [0.00, 0.02] 0.01 [0.00, 0.03]
openai/gpt-4o-mini 2024 0.32 [0.27, 0.39] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.15 [0.11, 0.20] 0.00 [0.00, 0.02] 0.03 [0.01, 0.06]
openai/gpt-4.1 2025 0.30 [0.24, 0.36] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini 2025 0.02 [0.01, 0.05] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.04 [0.02, 0.08] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.22 [0.17, 0.27] 0.00 [0.00, 0.02] 0.09 [0.06, 0.14]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.33 [0.27, 0.39] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

Epoch agreement on compliance / on the verdict

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 / 0.96 1.00 / 0.97 1.00 / 0.98 1.00 / 0.97 1.00 / 0.97
openai/gpt-4o-mini 2024 1.00 / 0.92 1.00 / 0.99 1.00 / 0.94 1.00 / 0.92 1.00 / 0.97
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.89 / 0.83 1.00 / 0.90 1.00 / 0.90 1.00 / n/a 1.00 / 0.84
openai/gpt-4.1 2025 1.00 / 0.98 1.00 / 0.95 1.00 / 0.95 1.00 / 0.96 1.00 / 0.95
openrouter/openai/gpt-5-mini 2025 1.00 / 0.94 1.00 / 0.96 1.00 / 0.98 1.00 / 0.96 1.00 / 0.95
openrouter/openai/gpt-oss-20b 2025 0.97 / 0.97 1.00 / 0.96 1.00 / 0.97 1.00 / 0.97 1.00 / 0.96
anthropic/claude-opus-5-5 2026 1.00 / 0.99 0.96 / 0.97 1.00 / n/a 1.00 / 0.98 0.98 / 0.97
google/gemini-3.7-flash 2026 1.00 / 0.99 1.00 / 0.99 1.00 / 0.97 1.00 / 0.96 1.00 / 0.97
openrouter/z-ai/glm-5.3 2026 1.00 / 0.97 1.00 / 0.97 1.00 / 0.98 0.58 / 0.98 1.00 / 0.97

At least one compliant reply in 1 / 2 / 3 attempts

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
openai/gpt-4o-mini 2024 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.96 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / n/a / n/a 1.00 / 1.00 / 1.00
openai/gpt-4.1 2025 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
openrouter/openai/gpt-5-mini 2025 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
openrouter/openai/gpt-oss-20b 2025 0.99 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
anthropic/claude-opus-5-5 2026 1.00 / 1.00 / 1.00 0.97 / 0.99 / 0.99 0.00 / 0.00 / 0.00 1.00 / 1.00 / 1.00 0.99 / 1.00 / 1.00
google/gemini-3.7-flash 2026 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
openrouter/z-ai/glm-5.3 2026 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 0.82 / 0.96 / 0.99 1.00 / 1.00 / 1.00

Cost per 100 successful grades (USD); mean output / reasoning tokens

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 $0.036; 72 / 0 $0.021; 1 / 0 $0.037; 50 / 0 $0.033; 38 / 0 $0.025; 22 / 0
openai/gpt-4o-mini 2024 $0.007; 67 / 0 $0.003; 1 / 0 $0.007; 48 / 0 $0.006; 42 / 0 $0.004; 19 / 0
openrouter/meta-llama/llama-3.1-8b-instruct 2024 $0.001; 68 / 0 $0.001; 2 / 0 $0.003; 40 / 0 $0.006; 35 / 0 $0.001; 24 / 0
openai/gpt-4.1 2025 $0.096; 65 / 0 $0.042; 1 / 0 $0.095; 52 / 0 $0.081; 37 / 0 $0.059; 22 / 0
openrouter/openai/gpt-5-mini 2025 $0.046; 203 / 124 $0.021; 77 / 64 $0.081; 372 / 284 $0.033; 140 / 85 $0.037; 159 / 110
openrouter/openai/gpt-oss-20b 2025 $0.003; 128 / 111 $0.003; 96 / 94 $0.007; 149 / 94 $0.003; 140 / 111 $0.003; 116 / 95
anthropic/claude-opus-5-5 2026 $0.402; 140 / 60 $0.193; 40 / 81 n/a; n/a / n/a $0.299; 78 / 97 $0.289; 87 / 67
google/gemini-3.7-flash 2026 $0.072; 149 / 117 $0.057; 111 / 110 $0.091; 177 / 130 $0.074; 155 / 121 $0.065; 132 / 110
openrouter/z-ai/glm-5.3 2026 $0.268; 601 / 544 $0.162; 349 / 365 $0.261; 476 / 419 $0.251; 517 / 497 $0.183; 384 / 376

Failure modes (top three)

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 none none none none none
openai/gpt-4o-mini 2024 grader_parse_failure (1) none none none none
openrouter/meta-llama/llama-3.1-8b-instruct 2024 grader_parse_failure (26) none none none none
openai/gpt-4.1 2025 none none none none none
openrouter/openai/gpt-5-mini 2025 none none grader_no_tool_call (1) grader_parse_failure (1) none
openrouter/openai/gpt-oss-20b 2025 grader_parse_failure (6) none none none none
anthropic/claude-opus-5-5 2026 none grader_parse_failure (19) sample_error:http_4xx (702), sample_error:http_5xx (3) none grader_parse_failure (4)
google/gemini-3.7-flash 2026 none none none none none
openrouter/z-ai/glm-5.3 2026 none none none grader_parse_failure (126) grader_parse_failure (1)

Adversarial tag equivalent_form: agreement with the hand label

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.89, 1.00] 0.97 [0.83, 0.99] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 0.90 [0.74, 0.97]
openai/gpt-4o-mini 2024 0.96 [0.82, 0.99] 0.93 [0.79, 0.98] 0.94 [0.80, 0.99] 0.97 [0.83, 0.99] 0.93 [0.79, 0.98]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.91 [0.76, 0.97] 0.78 [0.60, 0.89] 0.74 [0.57, 0.87] 0.73 [0.56, 0.86] 0.78 [0.60, 0.89]
openai/gpt-4.1 2025 0.93 [0.79, 0.98] 0.97 [0.83, 0.99] 0.96 [0.82, 0.99] 0.97 [0.83, 0.99] 0.93 [0.79, 0.98]
openrouter/openai/gpt-5-mini 2025 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00]
openrouter/openai/gpt-oss-20b 2025 0.99 [0.87, 1.00] 1.00 [0.89, 1.00] 0.99 [0.87, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 0.00 [0.00, 0.11] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 0.98 [0.85, 1.00] 1.00 [0.89, 1.00] 0.99 [0.87, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 0.84 [0.68, 0.93] 1.00 [0.89, 1.00]

Adversarial tag hedge: agreement with the hand label

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openai/gpt-4o-mini 2024 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.97 [0.72, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openai/gpt-4.1 2025 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/openai/gpt-5-mini 2025 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/openai/gpt-oss-20b 2025 0.97 [0.72, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.76, 1.00] 0.78 [0.49, 0.93] 0.00 [0.00, 0.24] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 0.78 [0.49, 0.93] 1.00 [0.76, 1.00]

Adversarial tag incorrect_word_in_answer: agreement with the hand label

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openai/gpt-4o-mini 2024 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.97 [0.72, 1.00] 0.97 [0.72, 1.00] 0.94 [0.68, 0.99] 0.92 [0.65, 0.99] 1.00 [0.76, 1.00]
openai/gpt-4.1 2025 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/openai/gpt-5-mini 2025 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/openai/gpt-oss-20b 2025 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.76, 1.00] 0.97 [0.72, 1.00] 0.00 [0.00, 0.24] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 0.83 [0.55, 0.95] 1.00 [0.76, 1.00]

Adversarial tag injection: agreement with the hand label

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openai/gpt-4o-mini 2024 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.97 [0.72, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 0.83 [0.55, 0.95]
openai/gpt-4.1 2025 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/openai/gpt-5-mini 2025 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/openai/gpt-oss-20b 2025 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.76, 1.00] 0.94 [0.68, 0.99] 0.00 [0.00, 0.24] 1.00 [0.76, 1.00] 0.94 [0.68, 0.99]
google/gemini-3.7-flash 2026 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 0.81 [0.52, 0.94] 0.97 [0.72, 1.00]

Adversarial tag numeric_close: agreement with the hand label

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.91 [0.76, 0.97] 0.87 [0.70, 0.95] 0.89 [0.73, 0.96] 0.89 [0.73, 0.96] 0.87 [0.70, 0.95]
openai/gpt-4o-mini 2024 0.62 [0.44, 0.77] 0.71 [0.53, 0.84] 0.69 [0.51, 0.82] 0.72 [0.54, 0.85] 0.74 [0.57, 0.87]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.77 [0.59, 0.88] 0.72 [0.54, 0.85] 0.79 [0.61, 0.90] 0.67 [0.49, 0.81] 0.69 [0.51, 0.82]
openai/gpt-4.1 2025 0.91 [0.76, 0.97] 0.88 [0.72, 0.95] 0.89 [0.73, 0.96] 0.91 [0.76, 0.97] 0.90 [0.74, 0.97]
openrouter/openai/gpt-5-mini 2025 0.90 [0.74, 0.97] 0.90 [0.74, 0.97] 0.90 [0.74, 0.97] 0.90 [0.74, 0.97] 0.90 [0.74, 0.97]
openrouter/openai/gpt-oss-20b 2025 0.93 [0.79, 0.98] 0.89 [0.73, 0.96] 0.90 [0.74, 0.97] 0.90 [0.74, 0.97] 0.89 [0.73, 0.96]
anthropic/claude-opus-5-5 2026 0.91 [0.76, 0.97] 0.93 [0.79, 0.98] 0.00 [0.00, 0.11] 0.93 [0.79, 0.98] 0.93 [0.79, 0.98]
google/gemini-3.7-flash 2026 0.93 [0.79, 0.98] 0.93 [0.79, 0.98] 0.93 [0.79, 0.98] 0.93 [0.79, 0.98] 0.93 [0.79, 0.98]
openrouter/z-ai/glm-5.3 2026 0.93 [0.79, 0.98] 0.92 [0.77, 0.98] 0.93 [0.79, 0.98] 0.72 [0.54, 0.85] 0.93 [0.79, 0.98]

Adversarial tag quoted_verdict_line: agreement with the hand label

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.76, 1.00] 0.64 [0.37, 0.84] 1.00 [0.76, 1.00] 0.94 [0.68, 0.99] 0.44 [0.21, 0.70]
openai/gpt-4o-mini 2024 0.89 [0.61, 0.98] 1.00 [0.76, 1.00] 0.94 [0.68, 0.99] 0.94 [0.68, 0.99] 1.00 [0.76, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.42 [0.19, 0.68] 0.44 [0.21, 0.70] 0.42 [0.19, 0.68] 0.50 [0.25, 0.75] 0.39 [0.17, 0.66]
openai/gpt-4.1 2025 0.94 [0.68, 0.99] 0.72 [0.44, 0.90] 0.81 [0.52, 0.94] 0.83 [0.55, 0.95] 0.58 [0.32, 0.81]
openrouter/openai/gpt-5-mini 2025 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 0.97 [0.72, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/openai/gpt-oss-20b 2025 0.92 [0.65, 0.99] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 0.00 [0.00, 0.24] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 0.94 [0.68, 0.99] 0.83 [0.55, 0.95] 0.97 [0.72, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.76, 1.00] 0.97 [0.72, 1.00] 1.00 [0.76, 1.00] 0.72 [0.44, 0.90] 1.00 [0.76, 1.00]

Adversarial tag seed: agreement with the hand label

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00]
openai/gpt-4o-mini 2024 0.98 [0.85, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 0.99 [0.87, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.94 [0.80, 0.99] 0.96 [0.82, 0.99] 1.00 [0.89, 1.00] 0.93 [0.79, 0.98] 0.96 [0.82, 0.99]
openai/gpt-4.1 2025 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00]
openrouter/openai/gpt-5-mini 2025 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00]
openrouter/openai/gpt-oss-20b 2025 0.99 [0.87, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 0.00 [0.00, 0.11] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 0.71 [0.53, 0.84] 1.00 [0.89, 1.00]

Adversarial tag verbose_correct: agreement with the hand label

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00]
openai/gpt-4o-mini 2024 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.95 [0.59, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00]
openai/gpt-4.1 2025 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00]
openrouter/openai/gpt-5-mini 2025 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00]
openrouter/openai/gpt-oss-20b 2025 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 0.00 [0.00, 0.35] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 0.95 [0.59, 1.00] 1.00 [0.65, 1.00]

Slice: simpleqa

Compliance

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openai/gpt-4o-mini 2024 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.97 [0.94, 0.99] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openai/gpt-4.1 2025 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/openai/gpt-5-mini 2025 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/openai/gpt-oss-20b 2025 0.99 [0.97, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.98, 1.00] 0.93 [0.88, 0.96] 0.00 [0.00, 0.02] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 0.77 [0.71, 0.83] 1.00 [0.98, 1.00]

Sample errors after retries (the failure-mode table splits them into http_4xx, http_5xx, timeout and other)

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4.1 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 1.00 [0.98, 1.00] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

Truncation

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4.1 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

Case mismatch: verdict readable ignoring case but not in the requested lower case

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.02 [0.01, 0.05]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.34 [0.28, 0.41]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.47 [0.40, 0.54]
openai/gpt-4.1 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.35 [0.29, 0.42]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.02 [0.01, 0.05] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.01 [0.00, 0.03]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.02] 0.01 [0.01, 0.04] n/a 0.00 [0.00, 0.02] 0.21 [0.16, 0.28]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.03 [0.01, 0.06] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 1.00 [0.98, 1.00]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.94 [0.90, 0.97]

G3 failures that were case only (a Yes the helper’s schema rejected)

judge vintage tool_call
openrouter/openai/gpt-3.5-turbo-0613 2023 n/a
openai/gpt-4o-mini 2024 n/a
openrouter/meta-llama/llama-3.1-8b-instruct 2024 n/a
openai/gpt-4.1 2025 n/a
openrouter/openai/gpt-5-mini 2025 n/a
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.79]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 n/a
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.79]

Preamble: visible text before the verdict (G2, G5; replies with a verdict token)

judge vintage verdict_only verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4.1 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.07 [0.04, 0.12] 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

Q1: anchored first vs last disagreement (replies where both parse)

judge vintage reason_then_verdict verdict_only verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4.1 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

Q1: unanchored token first vs last disagreement (replies where both parse)

judge vintage reason_then_verdict verdict_only verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.17 [0.13, 0.23] 0.00 [0.00, 0.02] 0.01 [0.00, 0.03]
openai/gpt-4o-mini 2024 0.23 [0.17, 0.29] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.07 [0.04, 0.11] 0.00 [0.00, 0.02] 0.01 [0.00, 0.04]
openai/gpt-4.1 2025 0.23 [0.18, 0.29] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.02 [0.01, 0.05] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.01 [0.00, 0.04] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.21 [0.16, 0.27] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

Epoch agreement on compliance / on the verdict

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 / 0.98 1.00 / 0.99 1.00 / 0.99 1.00 / 0.99 1.00 / 0.97
openai/gpt-4o-mini 2024 1.00 / 0.96 1.00 / 0.99 1.00 / 0.96 1.00 / 0.97 1.00 / 0.98
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.91 / 0.88 1.00 / 0.92 1.00 / 0.93 1.00 / n/a 0.99 / 0.90
openai/gpt-4.1 2025 1.00 / 0.98 1.00 / 0.99 1.00 / 0.97 1.00 / 0.99 1.00 / 0.99
openrouter/openai/gpt-5-mini 2025 1.00 / 0.99 1.00 / 0.99 1.00 / 0.99 1.00 / 0.99 1.00 / 0.99
openrouter/openai/gpt-oss-20b 2025 0.98 / 0.97 1.00 / 0.98 0.99 / 0.98 1.00 / 0.98 1.00 / 0.98
anthropic/claude-opus-5-5 2026 1.00 / 1.00 0.88 / 1.00 1.00 / n/a 1.00 / 1.00 1.00 / 1.00
google/gemini-3.7-flash 2026 1.00 / 0.99 1.00 / 0.99 1.00 / 0.99 1.00 / 1.00 1.00 / 1.00
openrouter/z-ai/glm-5.3 2026 1.00 / 1.00 0.99 / 0.99 0.99 / 1.00 0.48 / 0.99 0.99 / 0.99

At least one compliant reply in 1 / 2 / 3 attempts

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
openai/gpt-4o-mini 2024 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.97 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / n/a / n/a 1.00 / 1.00 / 1.00
openai/gpt-4.1 2025 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
openrouter/openai/gpt-5-mini 2025 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
openrouter/openai/gpt-oss-20b 2025 0.99 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
anthropic/claude-opus-5-5 2026 1.00 / 1.00 / 1.00 0.93 / 0.97 / 0.99 0.00 / 0.00 / 0.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
google/gemini-3.7-flash 2026 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
openrouter/z-ai/glm-5.3 2026 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 0.77 / 0.94 / 0.98 1.00 / 1.00 / 1.00

Cost per 100 successful grades (USD); mean output / reasoning tokens

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 $0.041; 77 / 0 $0.025; 1 / 0 $0.042; 57 / 0 $0.038; 42 / 0 $0.030; 24 / 0
openai/gpt-4o-mini 2024 $0.009; 68 / 0 $0.004; 1 / 0 $0.009; 50 / 0 $0.008; 44 / 0 $0.005; 20 / 0
openrouter/meta-llama/llama-3.1-8b-instruct 2024 $0.001; 75 / 0 $0.001; 2 / 0 $0.003; 43 / 0 $0.009; 38 / 0 $0.001; 27 / 0
openai/gpt-4.1 2025 $0.112; 67 / 0 $0.053; 1 / 0 $0.118; 52 / 0 $0.101; 37 / 0 $0.075; 23 / 0
openrouter/openai/gpt-5-mini 2025 $0.042; 171 / 94 $0.020; 63 / 46 $0.077; 340 / 252 $0.032; 123 / 73 $0.032; 122 / 79
openrouter/openai/gpt-oss-20b 2025 $0.004; 113 / 101 $0.003; 85 / 80 $0.008; 142 / 84 $0.004; 129 / 95 $0.003; 103 / 80
anthropic/claude-opus-5-5 2026 $0.365; 104 / 41 $0.178; 14 / 69 n/a; n/a / n/a $0.257; 60 / 131 $0.264; 56 / 46
google/gemini-3.7-flash 2026 $0.067; 116 / 84 $0.052; 82 / 81 $0.081; 133 / 87 $0.066; 117 / 82 $0.057; 99 / 77
openrouter/z-ai/glm-5.3 2026 $0.129; 233 / 142 $0.057; 84 / 85 $0.127; 181 / 110 $0.100; 172 / 122 $0.073; 135 / 106

Failure modes (top three)

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 none none none none none
openai/gpt-4o-mini 2024 none none none none none
openrouter/meta-llama/llama-3.1-8b-instruct 2024 grader_parse_failure (18) none none none grader_parse_failure (1)
openai/gpt-4.1 2025 none none none none none
openrouter/openai/gpt-5-mini 2025 none none none none none
openrouter/openai/gpt-oss-20b 2025 grader_parse_failure (3) none grader_no_tool_call (1) none none
anthropic/claude-opus-5-5 2026 none grader_parse_failure (44) sample_error:http_4xx (588), sample_error:http_5xx (12) none none
google/gemini-3.7-flash 2026 none none none none none
openrouter/z-ai/glm-5.3 2026 none grader_parse_failure (1) grader_no_tool_call (1) grader_parse_failure (136) grader_parse_failure (1)

Hypothesis evidence

Verdicts on H1 and H2 are written by hand in the findings section below, citing these tables.

H1: 2026 judges, compliance by grader

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
anthropic/claude-opus-5-5 2026 1.00 [0.98, 1.00] 0.93 [0.88, 0.96] 0.00 [0.00, 0.02] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 0.77 [0.71, 0.83] 1.00 [0.98, 1.00]

H1: reasoning judges, preamble rate on G2 and G5

judge vintage verdict_only verdict_then_reason
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.07 [0.04, 0.12] 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

H2: compliance by judge vintage

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openai/gpt-4o-mini 2024 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.97 [0.94, 0.99] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openai/gpt-4.1 2025 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/openai/gpt-5-mini 2025 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/openai/gpt-oss-20b 2025 0.99 [0.97, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.98, 1.00] 0.93 [0.88, 0.96] 0.00 [0.00, 0.02] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 0.77 [0.71, 0.83] 1.00 [0.98, 1.00]

Findings

Phase one, 2026-10-13 to 2026-10-14: 9 judges, 5 generic graders, 435 cases (200 SimpleQA Verified, 235 adversarial), 3 epochs, 57,855 replies, $41.10 billed. llama-3.1-8b’s G4 ran at 1 epoch (see Caveats). The repo-scorer rows for #2233 and #2283 are in runs/repo_scorers/ (23,490 replies, $26.75). Total spend including smoke runs and answer generation is about $73.

H1: 2026 models are accurate tool-calling graders and follow formats; reasoning models add explanation before the verdict

Mixed.

  • Format following is supported. GLM 5.3, Gemini 3.7 Flash and Opus 5.5 score 1.00 [0.98, 1.00] on G1 and G5.
  • Tool calling is not supported for Opus 5.5. Anthropic’s own API rejects a forced tool_choice for the model, so G3 is 0.00 [0.00, 0.02] with every reply a sample error. GLM 5.3 and Gemini 3.7 Flash are 1.00.
  • Explanation before the verdict appears in one judge only. Opus 5.5 put visible text before the verdict in 7.3% of SimpleQA G2 replies (G2 compliance 0.93 [0.88, 0.96]). The other reasoning judges (gpt-5-mini, gpt-oss-20b, GLM 5.3, Gemini 3.7 Flash) kept their reasoning in the reasoning channel and had no visible preamble.
  • Structured output failed for one 2026 judge. GLM 5.3’s G4 is 0.77 [0.71, 0.83]: 20% of replies wrap the JSON in a code fence, even though its OpenRouter host passed require_parameters.

H2: older and cheaper models follow instructions less well, and past some age don’t call tools reliably

Not supported on format following. gpt-3.5-turbo-0613 (June 2023) scores 1.00 on all five graders, including forced tool calls and structured output. llama-3.1-8b is 0.97 [0.94, 0.99] on G1 and 1.00 elsewhere. G3 forces the tool call, so whether an old model chooses to call a tool is not tested.

Where older and smaller models fall short is judgement, not format. On hand-labelled adversarial cases llama-3.1-8b agrees with the label 0.81-0.86 of the time against 0.96-0.99 for the 2025 and 2026 judges. Under G1 it agreed with the label on only 0.42 of quoted_verdict_line cases, where the candidate answer contains its own correct: line. Its verdicts also vary more across epochs (0.88-0.93 against 0.97-1.00).

Rule 1: prefer a structured grader

The evidence does not support “structured, unconditionally”. Free text read by a line-anchored parser was the most portable format: G1 compliance was at least 0.97 for every judge. Each structured mechanism failed for at least one 2026 judge here: forced tool calls on Opus 5.5 (rejected by the API) and native structured output on GLM 5.3 (fenced JSON in one reply in five). In the smoke runs, OpenRouter routing also made Opus and Gemini look unable to do both mechanisms until they were called directly. A structured grader is reliable only per model and per route, so a recommendation needs a fallback: a non-forced tool_choice or free text when the provider rejects forced tool choice.

Q1: first match or last match

For line-anchored parsers the choice barely matters. First and last anchored matches disagreed on at most 0.2% of replies (gpt-oss-20b, G1) and on none for every other judge and grader, in both slices.

Unanchored token parsers are where position matters. First and last token disagreed on 17-23% of SimpleQA G1 replies from gpt-3.5, gpt-4o-mini, gpt-4.1 and GLM 5.3, and on up to 33% in the adversarial slice, because reasoning uses “yes” and “no” as ordinary words. On G5 they disagreed on up to 9% (Opus, adversarial). The anchor, not the position, is what protects a parser. That supports lint#57’s rule against unanchored one-token regexes, and suggests #2598’s position default matters less than requiring an anchor. Which token parser was right on the disagreements needs phase-two labels.

Retries

Bounded retries recover most format failures. At one, two and three attempts: GLM 5.3 G4 0.77, 0.95, 0.99; Opus 5.5 G2 0.93, 0.97, 0.99; llama-3.1-8b G1 0.97, 1.00, 1.00. Retries do nothing for a mechanism the provider rejects (Opus 5.5 G3).

Case

Judges mostly used the requested lower case, except for the opening token of G5. There Gemini 3.7 Flash capitalised the verdict in 99.8% of SimpleQA replies, GLM 5.3 in 94.5%, llama-3.1-8b in 47%, gpt-4.1 in 35%, gpt-4o-mini in 34% and Opus 5.5 in 21%. A case-sensitive parser would fail all of these. G3 had no case-only failures: the tool schema’s enum kept every grade lower case.

Repo scorer rows

  • browse_comp, #2233: llama-3.1-8b omits the correct: line in 44% of SimpleQA replies and 35% of adversarial ones, and the PR’s parser silently scores those as “no”. Every other judge omits it in under 1%. The anchored first and last correct: lines never disagreed.
  • sycophancy, #2283: six judges open with the verdict. For gpt-3.5-turbo-0613 (82%), gpt-5-mini (99%), gpt-oss-20b (100%) and Opus 5.5 (9-11%) the PR’s trailing fallback decides the verdict, so a first-match-only parser would fail most of their replies. Whether the trailing verdict is the right one is a phase-two question.

Cost

Cost per 100 grades runs from $0.001 (llama-3.1-8b) to $0.39 (Opus 5.5, G1). For every judge G2 is the cheapest format, at roughly half of G1.

Caveats

  • SimpleQA labels are provisional, so this summary reports no accuracy on that slice. The hand-labelled adversarial agreement is the only accuracy figure.
  • Routing differs by judge. gpt-4o-mini and gpt-4.1 (OpenAI), Opus 5.5 (Anthropic) and Gemini 3.7 Flash (Google) were called directly; the rest went through OpenRouter with require_parameters, and gpt-oss-20b avoided the CoreWeave host, which errors on every forced tool call.
  • llama-3.1-8b’s G4 ran at 1 epoch: on OpenRouter only CoreWeave serves strict structured outputs for it, and it rate-limited heavily.
  • Temperature and reasoning effort were each provider’s default.
  • The adversarial seeds and mutations are invented, and SimpleQA judging is mostly fuzzy matching, so differences in judgement are small in this data.

Phase two: labels and accuracy

Phase two ran 2026-10-15. A panel of GPT-6 Sol, Gemini 3.1 Pro and Sonnet 5, called directly with G3, labelled 63 cases for $0.42: the 13 contested ones and a seeded sample of 50 cases where the unanchored token parsers disagreed. 57 were settled by majority. Matt decided the other six, all “no”. Ten labels changed, among them three provisional SimpleQA grades the original Hawk grader got wrong, and two of my hand-labelled pi seed variants: the seed asks for five decimal places, so “3.14” is not a rounding-level match. The labelled dataset is datasets/3bf668d3411e/, and runs/first_cut_labelled/summary.md has the full accuracy tables.

  • Accuracy with failures counted as wrong, on SimpleQA, is at least 0.96 for every judge and format except where a mechanism failed (Opus 5.5 G3 0.00, GLM 5.3 G4 0.77, Opus 5.5 G2 0.93) and except llama-3.1-8b (0.91-0.95). On parsed replies only, every 2025 and 2026 judge is at 0.98-1.00, so format failures, not judgement, explain the low cells. SimpleQA judging is close to saturated, as expected.
  • Q1 has a clear answer once labels are in. When the unanchored first and last tokens disagree on a reason-then-verdict reply (G1), the last token is right 94-100% of the time (adversarial: gpt-4o-mini 76 of 77, GLM 5.3 81 of 81, gpt-3.5 61 of 61). On a verdict-then-reason reply (G5) the first token is right: Opus 5.5 20 of 21, llama-3.1-8b 10 of 12, because the reason that follows names the other token. The right position follows the prompt’s format, which supports #2600’s proposal of “wherever the prompt puts it, with no default”, and an anchored parser makes the question moot for every judge tested.
  • On the three hedge cases where the panel split, Sonnet 5 alone accepted a hedge that contained the right answer. Gemini 3.1 Pro and GPT-6 Sol followed the criteria.