Grader comparison: first_cut

Dataset 8626229b0d04e1329516c066ca53f89784c048d3400460c0ee2a039b24443173. Git commit 3b5115c3d6c2b7d73d95a631a37433aaf51563e6. inspect_ai 0.3.263.

Rates are per-case means across epochs, with 95% Wilson intervals over cases; ratios use a bootstrap over cases. Each cell is rate [low, high].

G3 (tool_call) forces the tool call with tool_choice, so its compliance measures the repo’s helper as used, not whether a model chooses to call a tool. Every grader reads verdicts case-insensitively except G3, whose helper validates the grade before upper-casing it; the case tables show how much that costs. Inspect’s OpenAI-compatible provider, which OpenRouter uses, parses a tool call written as text into a real one only under the emulate_tools model arg, which this harness does not set, so G3’s no-tool-call rate has no such rescue.

Slice: adversarial

Compliance

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openai/gpt-4o-mini 2024 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.96 [0.93, 0.98] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openai/gpt-4.1 2025 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/openai/gpt-5-mini 2025 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/openai/gpt-oss-20b 2025 0.99 [0.97, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.98, 1.00] 0.97 [0.94, 0.99] 0.00 [0.00, 0.02] 1.00 [0.98, 1.00] 0.99 [0.97, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 0.82 [0.77, 0.86] 1.00 [0.98, 1.00]

Sample errors after retries (the failure-mode table splits them into http_4xx, http_5xx, timeout and other)

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4.1 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 1.00 [0.98, 1.00] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

Truncation

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4.1 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

Case mismatch: verdict readable ignoring case but not in the requested lower case

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.02 [0.01, 0.04]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.39 [0.33, 0.46]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.02] 0.02 [0.01, 0.04] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.59 [0.53, 0.65]
openai/gpt-4.1 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.22 [0.17, 0.28]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.02 [0.01, 0.04] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.02] 0.02 [0.01, 0.05] n/a 0.00 [0.00, 0.02] 0.11 [0.08, 0.16]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.01 [0.00, 0.03] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.99 [0.97, 1.00]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.91 [0.87, 0.94]

G3 failures that were case only (a Yes the helper’s schema rejected)

judge vintage tool_call
openrouter/openai/gpt-3.5-turbo-0613 2023 n/a
openai/gpt-4o-mini 2024 n/a
openrouter/meta-llama/llama-3.1-8b-instruct 2024 n/a
openai/gpt-4.1 2025 n/a
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.79]
openrouter/openai/gpt-oss-20b 2025 n/a
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 n/a
openrouter/z-ai/glm-5.3 2026 n/a

Preamble: visible text before the verdict (G2, G5; replies with a verdict token)

judge vintage verdict_only verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4.1 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.03 [0.01, 0.06] 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

Q1: anchored first vs last disagreement (replies where both parse)

judge vintage reason_then_verdict verdict_only verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4.1 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

Q1: unanchored token first vs last disagreement (replies where both parse)

judge vintage reason_then_verdict verdict_only verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.28 [0.23, 0.34] 0.00 [0.00, 0.02] 0.01 [0.00, 0.03]
openai/gpt-4o-mini 2024 0.32 [0.27, 0.39] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.15 [0.11, 0.20] 0.00 [0.00, 0.02] 0.03 [0.01, 0.06]
openai/gpt-4.1 2025 0.30 [0.24, 0.36] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini 2025 0.02 [0.01, 0.05] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.04 [0.02, 0.08] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.22 [0.17, 0.27] 0.00 [0.00, 0.02] 0.09 [0.06, 0.14]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.33 [0.27, 0.39] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

Epoch agreement on compliance / on the verdict

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 / 0.96 1.00 / 0.97 1.00 / 0.98 1.00 / 0.97 1.00 / 0.97
openai/gpt-4o-mini 2024 1.00 / 0.92 1.00 / 0.99 1.00 / 0.94 1.00 / 0.92 1.00 / 0.97
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.89 / 0.83 1.00 / 0.90 1.00 / 0.90 1.00 / n/a 1.00 / 0.84
openai/gpt-4.1 2025 1.00 / 0.98 1.00 / 0.95 1.00 / 0.95 1.00 / 0.96 1.00 / 0.95
openrouter/openai/gpt-5-mini 2025 1.00 / 0.94 1.00 / 0.96 1.00 / 0.98 1.00 / 0.96 1.00 / 0.95
openrouter/openai/gpt-oss-20b 2025 0.97 / 0.97 1.00 / 0.96 1.00 / 0.97 1.00 / 0.97 1.00 / 0.96
anthropic/claude-opus-5-5 2026 1.00 / 0.99 0.96 / 0.97 1.00 / n/a 1.00 / 0.98 0.98 / 0.97
google/gemini-3.7-flash 2026 1.00 / 0.99 1.00 / 0.99 1.00 / 0.97 1.00 / 0.96 1.00 / 0.97
openrouter/z-ai/glm-5.3 2026 1.00 / 0.97 1.00 / 0.97 1.00 / 0.98 0.58 / 0.98 1.00 / 0.97

At least one compliant reply in 1 / 2 / 3 attempts

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
openai/gpt-4o-mini 2024 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.96 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / n/a / n/a 1.00 / 1.00 / 1.00
openai/gpt-4.1 2025 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
openrouter/openai/gpt-5-mini 2025 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
openrouter/openai/gpt-oss-20b 2025 0.99 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
anthropic/claude-opus-5-5 2026 1.00 / 1.00 / 1.00 0.97 / 0.99 / 0.99 0.00 / 0.00 / 0.00 1.00 / 1.00 / 1.00 0.99 / 1.00 / 1.00
google/gemini-3.7-flash 2026 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
openrouter/z-ai/glm-5.3 2026 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 0.82 / 0.96 / 0.99 1.00 / 1.00 / 1.00

Cost per 100 successful grades (USD); mean output / reasoning tokens

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 $0.036; 72 / 0 $0.021; 1 / 0 $0.037; 50 / 0 $0.033; 38 / 0 $0.025; 22 / 0
openai/gpt-4o-mini 2024 $0.007; 67 / 0 $0.003; 1 / 0 $0.007; 48 / 0 $0.006; 42 / 0 $0.004; 19 / 0
openrouter/meta-llama/llama-3.1-8b-instruct 2024 $0.001; 68 / 0 $0.001; 2 / 0 $0.003; 40 / 0 $0.006; 35 / 0 $0.001; 24 / 0
openai/gpt-4.1 2025 $0.096; 65 / 0 $0.042; 1 / 0 $0.095; 52 / 0 $0.081; 37 / 0 $0.059; 22 / 0
openrouter/openai/gpt-5-mini 2025 $0.046; 203 / 124 $0.021; 77 / 64 $0.081; 372 / 284 $0.033; 140 / 85 $0.037; 159 / 110
openrouter/openai/gpt-oss-20b 2025 $0.003; 128 / 111 $0.003; 96 / 94 $0.007; 149 / 94 $0.003; 140 / 111 $0.003; 116 / 95
anthropic/claude-opus-5-5 2026 $0.402; 140 / 60 $0.193; 40 / 81 n/a; n/a / n/a $0.299; 78 / 97 $0.289; 87 / 67
google/gemini-3.7-flash 2026 $0.072; 149 / 117 $0.057; 111 / 110 $0.091; 177 / 130 $0.074; 155 / 121 $0.065; 132 / 110
openrouter/z-ai/glm-5.3 2026 $0.268; 601 / 544 $0.162; 349 / 365 $0.261; 476 / 419 $0.251; 517 / 497 $0.183; 384 / 376

Failure modes (top three)

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 none none none none none
openai/gpt-4o-mini 2024 grader_parse_failure (1) none none none none
openrouter/meta-llama/llama-3.1-8b-instruct 2024 grader_parse_failure (26) none none none none
openai/gpt-4.1 2025 none none none none none
openrouter/openai/gpt-5-mini 2025 none none grader_no_tool_call (1) grader_parse_failure (1) none
openrouter/openai/gpt-oss-20b 2025 grader_parse_failure (6) none none none none
anthropic/claude-opus-5-5 2026 none grader_parse_failure (19) sample_error:http_4xx (702), sample_error:http_5xx (3) none grader_parse_failure (4)
google/gemini-3.7-flash 2026 none none none none none
openrouter/z-ai/glm-5.3 2026 none none none grader_parse_failure (126) grader_parse_failure (1)

Adversarial tag equivalent_form: agreement with the hand label

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.89, 1.00] 0.97 [0.83, 0.99] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 0.90 [0.74, 0.97]
openai/gpt-4o-mini 2024 0.96 [0.82, 0.99] 0.93 [0.79, 0.98] 0.94 [0.80, 0.99] 0.97 [0.83, 0.99] 0.93 [0.79, 0.98]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.91 [0.76, 0.97] 0.78 [0.60, 0.89] 0.74 [0.57, 0.87] 0.73 [0.56, 0.86] 0.78 [0.60, 0.89]
openai/gpt-4.1 2025 0.93 [0.79, 0.98] 0.97 [0.83, 0.99] 0.96 [0.82, 0.99] 0.97 [0.83, 0.99] 0.93 [0.79, 0.98]
openrouter/openai/gpt-5-mini 2025 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00]
openrouter/openai/gpt-oss-20b 2025 0.99 [0.87, 1.00] 1.00 [0.89, 1.00] 0.99 [0.87, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 0.00 [0.00, 0.11] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 0.98 [0.85, 1.00] 1.00 [0.89, 1.00] 0.99 [0.87, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 0.84 [0.68, 0.93] 1.00 [0.89, 1.00]

Adversarial tag hedge: agreement with the hand label

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openai/gpt-4o-mini 2024 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.97 [0.72, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openai/gpt-4.1 2025 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/openai/gpt-5-mini 2025 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/openai/gpt-oss-20b 2025 0.97 [0.72, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.76, 1.00] 0.78 [0.49, 0.93] 0.00 [0.00, 0.24] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 0.78 [0.49, 0.93] 1.00 [0.76, 1.00]

Adversarial tag incorrect_word_in_answer: agreement with the hand label

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openai/gpt-4o-mini 2024 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.97 [0.72, 1.00] 0.97 [0.72, 1.00] 0.94 [0.68, 0.99] 0.92 [0.65, 0.99] 1.00 [0.76, 1.00]
openai/gpt-4.1 2025 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/openai/gpt-5-mini 2025 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/openai/gpt-oss-20b 2025 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.76, 1.00] 0.97 [0.72, 1.00] 0.00 [0.00, 0.24] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 0.83 [0.55, 0.95] 1.00 [0.76, 1.00]

Adversarial tag injection: agreement with the hand label

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openai/gpt-4o-mini 2024 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.97 [0.72, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 0.83 [0.55, 0.95]
openai/gpt-4.1 2025 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/openai/gpt-5-mini 2025 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/openai/gpt-oss-20b 2025 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.76, 1.00] 0.94 [0.68, 0.99] 0.00 [0.00, 0.24] 1.00 [0.76, 1.00] 0.94 [0.68, 0.99]
google/gemini-3.7-flash 2026 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 0.81 [0.52, 0.94] 0.97 [0.72, 1.00]

Adversarial tag numeric_close: agreement with the hand label

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.98 [0.84, 1.00] 0.93 [0.77, 0.98] 0.95 [0.81, 0.99] 0.95 [0.81, 0.99] 0.93 [0.77, 0.98]
openai/gpt-4o-mini 2024 0.67 [0.48, 0.81] 0.76 [0.58, 0.88] 0.74 [0.55, 0.86] 0.77 [0.59, 0.89] 0.80 [0.62, 0.91]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.80 [0.62, 0.91] 0.77 [0.59, 0.89] 0.85 [0.67, 0.94] 0.71 [0.53, 0.85] 0.73 [0.54, 0.86]
openai/gpt-4.1 2025 0.98 [0.84, 1.00] 0.94 [0.79, 0.99] 0.95 [0.81, 0.99] 0.98 [0.84, 1.00] 0.96 [0.82, 0.99]
openrouter/openai/gpt-5-mini 2025 0.96 [0.82, 0.99] 0.96 [0.82, 0.99] 0.96 [0.82, 0.99] 0.96 [0.82, 0.99] 0.96 [0.82, 0.99]
openrouter/openai/gpt-oss-20b 2025 1.00 [0.88, 1.00] 0.95 [0.81, 0.99] 0.96 [0.82, 0.99] 0.96 [0.82, 0.99] 0.95 [0.81, 0.99]
anthropic/claude-opus-5-5 2026 0.98 [0.84, 1.00] 1.00 [0.88, 1.00] 0.00 [0.00, 0.12] 1.00 [0.88, 1.00] 1.00 [0.88, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.88, 1.00] 1.00 [0.88, 1.00] 1.00 [0.88, 1.00] 1.00 [0.88, 1.00] 1.00 [0.88, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.88, 1.00] 0.99 [0.86, 1.00] 1.00 [0.88, 1.00] 0.77 [0.59, 0.89] 1.00 [0.88, 1.00]

Adversarial tag quoted_verdict_line: agreement with the hand label

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.76, 1.00] 0.64 [0.37, 0.84] 1.00 [0.76, 1.00] 0.94 [0.68, 0.99] 0.44 [0.21, 0.70]
openai/gpt-4o-mini 2024 0.89 [0.61, 0.98] 1.00 [0.76, 1.00] 0.94 [0.68, 0.99] 0.94 [0.68, 0.99] 1.00 [0.76, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.42 [0.19, 0.68] 0.44 [0.21, 0.70] 0.42 [0.19, 0.68] 0.50 [0.25, 0.75] 0.39 [0.17, 0.66]
openai/gpt-4.1 2025 0.94 [0.68, 0.99] 0.72 [0.44, 0.90] 0.81 [0.52, 0.94] 0.83 [0.55, 0.95] 0.58 [0.32, 0.81]
openrouter/openai/gpt-5-mini 2025 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 0.97 [0.72, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
openrouter/openai/gpt-oss-20b 2025 0.92 [0.65, 0.99] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 0.00 [0.00, 0.24] 1.00 [0.76, 1.00] 1.00 [0.76, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.76, 1.00] 1.00 [0.76, 1.00] 0.94 [0.68, 0.99] 0.83 [0.55, 0.95] 0.97 [0.72, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.76, 1.00] 0.97 [0.72, 1.00] 1.00 [0.76, 1.00] 0.72 [0.44, 0.90] 1.00 [0.76, 1.00]

Adversarial tag seed: agreement with the hand label

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00]
openai/gpt-4o-mini 2024 0.98 [0.85, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 0.99 [0.87, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.94 [0.80, 0.99] 0.96 [0.82, 0.99] 1.00 [0.89, 1.00] 0.93 [0.79, 0.98] 0.96 [0.82, 0.99]
openai/gpt-4.1 2025 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00]
openrouter/openai/gpt-5-mini 2025 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00]
openrouter/openai/gpt-oss-20b 2025 0.99 [0.87, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 0.00 [0.00, 0.11] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 1.00 [0.89, 1.00] 0.71 [0.53, 0.84] 1.00 [0.89, 1.00]

Adversarial tag verbose_correct: agreement with the hand label

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00]
openai/gpt-4o-mini 2024 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.95 [0.59, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00]
openai/gpt-4.1 2025 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00]
openrouter/openai/gpt-5-mini 2025 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00]
openrouter/openai/gpt-oss-20b 2025 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 0.00 [0.00, 0.35] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 1.00 [0.65, 1.00] 0.95 [0.59, 1.00] 1.00 [0.65, 1.00]

Accuracy

Cases by label provenance: hand (143), original_grade (71), majority (15), reviewed (6)

Accuracy, failures counted as wrong

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.94 [0.91, 0.97] 0.95 [0.91, 0.97] 0.93 [0.89, 0.96] 0.92 [0.88, 0.95] 0.92 [0.88, 0.95]
openai/gpt-4o-mini 2024 0.90 [0.86, 0.94] 0.94 [0.91, 0.97] 0.93 [0.89, 0.96] 0.93 [0.89, 0.96] 0.94 [0.91, 0.97]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.82 [0.77, 0.87] 0.84 [0.79, 0.88] 0.85 [0.79, 0.89] 0.82 [0.76, 0.86] 0.81 [0.75, 0.85]
openai/gpt-4.1 2025 0.97 [0.94, 0.99] 0.95 [0.91, 0.97] 0.96 [0.93, 0.98] 0.96 [0.93, 0.98] 0.94 [0.91, 0.97]
openrouter/openai/gpt-5-mini 2025 0.97 [0.94, 0.98] 0.96 [0.93, 0.98] 0.98 [0.96, 0.99] 0.97 [0.94, 0.99] 0.96 [0.93, 0.98]
openrouter/openai/gpt-oss-20b 2025 0.97 [0.94, 0.99] 0.98 [0.95, 0.99] 0.98 [0.95, 0.99] 0.97 [0.94, 0.99] 0.97 [0.94, 0.98]
anthropic/claude-opus-5-5 2026 0.94 [0.90, 0.96] 0.90 [0.85, 0.93] 0.00 [0.00, 0.02] 0.95 [0.91, 0.97] 0.92 [0.88, 0.95]
google/gemini-3.7-flash 2026 0.99 [0.97, 1.00] 0.99 [0.97, 1.00] 0.99 [0.96, 0.99] 0.98 [0.95, 0.99] 0.99 [0.97, 1.00]
openrouter/z-ai/glm-5.3 2026 0.98 [0.96, 0.99] 0.98 [0.96, 0.99] 0.99 [0.96, 0.99] 0.81 [0.75, 0.85] 0.98 [0.95, 0.99]

Accuracy on parsed replies only

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.94 [0.92, 0.97] 0.95 [0.92, 0.97] 0.93 [0.90, 0.96] 0.92 [0.89, 0.96] 0.92 [0.89, 0.95]
openai/gpt-4o-mini 2024 0.90 [0.87, 0.94] 0.94 [0.91, 0.97] 0.93 [0.90, 0.96] 0.93 [0.90, 0.95] 0.94 [0.91, 0.97]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.86 [0.82, 0.89] 0.84 [0.80, 0.88] 0.85 [0.80, 0.88] 0.82 [0.77, 0.86] 0.81 [0.76, 0.85]
openai/gpt-4.1 2025 0.97 [0.95, 0.99] 0.95 [0.92, 0.97] 0.96 [0.94, 0.98] 0.96 [0.94, 0.98] 0.94 [0.92, 0.97]
openrouter/openai/gpt-5-mini 2025 0.97 [0.95, 0.98] 0.96 [0.94, 0.98] 0.99 [0.97, 1.00] 0.97 [0.95, 0.99] 0.96 [0.94, 0.98]
openrouter/openai/gpt-oss-20b 2025 0.98 [0.96, 0.99] 0.98 [0.96, 0.99] 0.98 [0.96, 0.99] 0.97 [0.95, 0.99] 0.97 [0.95, 0.98]
anthropic/claude-opus-5-5 2026 0.94 [0.90, 0.97] 0.92 [0.89, 0.96] n/a 0.95 [0.92, 0.98] 0.93 [0.90, 0.96]
google/gemini-3.7-flash 2026 0.99 [0.98, 1.00] 0.99 [0.98, 1.00] 0.99 [0.97, 1.00] 0.98 [0.96, 0.99] 0.99 [0.98, 1.00]
openrouter/z-ai/glm-5.3 2026 0.98 [0.97, 0.99] 0.98 [0.97, 0.99] 0.99 [0.97, 1.00] 0.98 [0.96, 0.99] 0.98 [0.96, 0.99]

Which parser was right when first_anchored and last_anchored disagreed (share right: first / last)

judge vintage reason_then_verdict
openrouter/openai/gpt-oss-20b 2025 first 0.00 [0.00, 0.79] / last 1.00 [0.21, 1.00] (cases=1)

Which parser was right when first_token and last_token disagreed (share right: first / last)

judge vintage reason_then_verdict verdict_only verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 first 0.00 [0.00, 0.06] / last 1.00 [0.94, 1.00] (cases=61) first 0.00 [0.00, 0.56] / last 1.00 [0.44, 1.00] (cases=3)
openai/gpt-4o-mini 2024 first 0.01 [0.00, 0.07] / last 0.99 [0.93, 1.00] (cases=77)
openrouter/meta-llama/llama-3.1-8b-instruct 2024 first 0.04 [0.01, 0.13] / last 0.96 [0.87, 0.99] (cases=52) first 0.83 [0.55, 0.95] / last 0.17 [0.05, 0.45] (cases=12)
openai/gpt-4.1 2025 first 0.00 [0.00, 0.05] / last 1.00 [0.95, 1.00] (cases=70) first 0.00 [0.00, 0.79] / last 1.00 [0.21, 1.00] (cases=1)
openrouter/openai/gpt-5-mini 2025 first 0.00 [0.00, 0.26] / last 1.00 [0.74, 1.00] (cases=11)
openrouter/openai/gpt-oss-20b 2025 first 0.00 [0.00, 0.13] / last 1.00 [0.87, 1.00] (cases=25) first 1.00 [0.21, 1.00] / last 0.00 [0.00, 0.79] (cases=1)
anthropic/claude-opus-5-5 2026 first 0.06 [0.02, 0.16] / last 0.94 [0.84, 0.98] (cases=52) first 0.00 [0.00, 0.79] / last 1.00 [0.21, 1.00] (cases=1) first 0.95 [0.77, 0.99] / last 0.05 [0.01, 0.23] (cases=21)
openrouter/z-ai/glm-5.3 2026 first 0.00 [0.00, 0.05] / last 1.00 [0.95, 1.00] (cases=81)

Slice: simpleqa

Compliance

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openai/gpt-4o-mini 2024 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.97 [0.94, 0.99] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openai/gpt-4.1 2025 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/openai/gpt-5-mini 2025 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/openai/gpt-oss-20b 2025 0.99 [0.97, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.98, 1.00] 0.93 [0.88, 0.96] 0.00 [0.00, 0.02] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 0.77 [0.71, 0.83] 1.00 [0.98, 1.00]

Sample errors after retries (the failure-mode table splits them into http_4xx, http_5xx, timeout and other)

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4.1 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 1.00 [0.98, 1.00] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

Truncation

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4.1 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

Case mismatch: verdict readable ignoring case but not in the requested lower case

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.02 [0.01, 0.05]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.34 [0.28, 0.41]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.47 [0.40, 0.54]
openai/gpt-4.1 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.35 [0.29, 0.42]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.02 [0.01, 0.05] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.01 [0.00, 0.03]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.02] 0.01 [0.01, 0.04] n/a 0.00 [0.00, 0.02] 0.21 [0.16, 0.28]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.03 [0.01, 0.06] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 1.00 [0.98, 1.00]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.94 [0.90, 0.97]

G3 failures that were case only (a Yes the helper’s schema rejected)

judge vintage tool_call
openrouter/openai/gpt-3.5-turbo-0613 2023 n/a
openai/gpt-4o-mini 2024 n/a
openrouter/meta-llama/llama-3.1-8b-instruct 2024 n/a
openai/gpt-4.1 2025 n/a
openrouter/openai/gpt-5-mini 2025 n/a
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.79]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 n/a
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.79]

Preamble: visible text before the verdict (G2, G5; replies with a verdict token)

judge vintage verdict_only verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4.1 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.07 [0.04, 0.12] 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

Q1: anchored first vs last disagreement (replies where both parse)

judge vintage reason_then_verdict verdict_only verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openai/gpt-4.1 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

Q1: unanchored token first vs last disagreement (replies where both parse)

judge vintage reason_then_verdict verdict_only verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.17 [0.13, 0.23] 0.00 [0.00, 0.02] 0.01 [0.00, 0.03]
openai/gpt-4o-mini 2024 0.23 [0.17, 0.29] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.07 [0.04, 0.11] 0.00 [0.00, 0.02] 0.01 [0.00, 0.04]
openai/gpt-4.1 2025 0.23 [0.18, 0.29] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.02 [0.01, 0.05] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.01 [0.00, 0.04] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.21 [0.16, 0.27] 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

Epoch agreement on compliance / on the verdict

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 / 0.98 1.00 / 0.99 1.00 / 0.99 1.00 / 0.99 1.00 / 0.97
openai/gpt-4o-mini 2024 1.00 / 0.96 1.00 / 0.99 1.00 / 0.96 1.00 / 0.97 1.00 / 0.98
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.91 / 0.88 1.00 / 0.92 1.00 / 0.93 1.00 / n/a 0.99 / 0.90
openai/gpt-4.1 2025 1.00 / 0.98 1.00 / 0.99 1.00 / 0.97 1.00 / 0.99 1.00 / 0.99
openrouter/openai/gpt-5-mini 2025 1.00 / 0.99 1.00 / 0.99 1.00 / 0.99 1.00 / 0.99 1.00 / 0.99
openrouter/openai/gpt-oss-20b 2025 0.98 / 0.97 1.00 / 0.98 0.99 / 0.98 1.00 / 0.98 1.00 / 0.98
anthropic/claude-opus-5-5 2026 1.00 / 1.00 0.88 / 1.00 1.00 / n/a 1.00 / 1.00 1.00 / 1.00
google/gemini-3.7-flash 2026 1.00 / 0.99 1.00 / 0.99 1.00 / 0.99 1.00 / 1.00 1.00 / 1.00
openrouter/z-ai/glm-5.3 2026 1.00 / 1.00 0.99 / 0.99 0.99 / 1.00 0.48 / 0.99 0.99 / 0.99

At least one compliant reply in 1 / 2 / 3 attempts

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
openai/gpt-4o-mini 2024 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.97 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / n/a / n/a 1.00 / 1.00 / 1.00
openai/gpt-4.1 2025 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
openrouter/openai/gpt-5-mini 2025 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
openrouter/openai/gpt-oss-20b 2025 0.99 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
anthropic/claude-opus-5-5 2026 1.00 / 1.00 / 1.00 0.93 / 0.97 / 0.99 0.00 / 0.00 / 0.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
google/gemini-3.7-flash 2026 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00
openrouter/z-ai/glm-5.3 2026 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 1.00 / 1.00 / 1.00 0.77 / 0.94 / 0.98 1.00 / 1.00 / 1.00

Cost per 100 successful grades (USD); mean output / reasoning tokens

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 $0.041; 77 / 0 $0.025; 1 / 0 $0.042; 57 / 0 $0.038; 42 / 0 $0.030; 24 / 0
openai/gpt-4o-mini 2024 $0.009; 68 / 0 $0.004; 1 / 0 $0.009; 50 / 0 $0.008; 44 / 0 $0.005; 20 / 0
openrouter/meta-llama/llama-3.1-8b-instruct 2024 $0.001; 75 / 0 $0.001; 2 / 0 $0.003; 43 / 0 $0.009; 38 / 0 $0.001; 27 / 0
openai/gpt-4.1 2025 $0.112; 67 / 0 $0.053; 1 / 0 $0.118; 52 / 0 $0.101; 37 / 0 $0.075; 23 / 0
openrouter/openai/gpt-5-mini 2025 $0.042; 171 / 94 $0.020; 63 / 46 $0.077; 340 / 252 $0.032; 123 / 73 $0.032; 122 / 79
openrouter/openai/gpt-oss-20b 2025 $0.004; 113 / 101 $0.003; 85 / 80 $0.008; 142 / 84 $0.004; 129 / 95 $0.003; 103 / 80
anthropic/claude-opus-5-5 2026 $0.365; 104 / 41 $0.178; 14 / 69 n/a; n/a / n/a $0.257; 60 / 131 $0.264; 56 / 46
google/gemini-3.7-flash 2026 $0.067; 116 / 84 $0.052; 82 / 81 $0.081; 133 / 87 $0.066; 117 / 82 $0.057; 99 / 77
openrouter/z-ai/glm-5.3 2026 $0.129; 233 / 142 $0.057; 84 / 85 $0.127; 181 / 110 $0.100; 172 / 122 $0.073; 135 / 106

Failure modes (top three)

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 none none none none none
openai/gpt-4o-mini 2024 none none none none none
openrouter/meta-llama/llama-3.1-8b-instruct 2024 grader_parse_failure (18) none none none grader_parse_failure (1)
openai/gpt-4.1 2025 none none none none none
openrouter/openai/gpt-5-mini 2025 none none none none none
openrouter/openai/gpt-oss-20b 2025 grader_parse_failure (3) none grader_no_tool_call (1) none none
anthropic/claude-opus-5-5 2026 none grader_parse_failure (44) sample_error:http_4xx (588), sample_error:http_5xx (12) none none
google/gemini-3.7-flash 2026 none none none none none
openrouter/z-ai/glm-5.3 2026 none grader_parse_failure (1) grader_no_tool_call (1) grader_parse_failure (136) grader_parse_failure (1)

Accuracy

Cases by label provenance: original_grade (179), majority (21)

Accuracy, failures counted as wrong

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.98 [0.95, 0.99] 0.98 [0.95, 0.99] 0.98 [0.95, 0.99] 0.98 [0.95, 0.99] 0.97 [0.94, 0.99]
openai/gpt-4o-mini 2024 0.96 [0.93, 0.98] 0.97 [0.93, 0.99] 0.96 [0.93, 0.98] 0.97 [0.93, 0.98] 0.98 [0.95, 0.99]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.91 [0.86, 0.94] 0.95 [0.91, 0.97] 0.95 [0.91, 0.97] 0.93 [0.88, 0.95] 0.94 [0.90, 0.96]
openai/gpt-4.1 2025 0.99 [0.96, 1.00] 0.98 [0.95, 0.99] 0.98 [0.95, 0.99] 0.98 [0.95, 0.99] 0.99 [0.97, 1.00]
openrouter/openai/gpt-5-mini 2025 0.99 [0.97, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 0.99 [0.97, 1.00]
openrouter/openai/gpt-oss-20b 2025 0.98 [0.95, 0.99] 0.98 [0.96, 0.99] 0.98 [0.95, 0.99] 0.98 [0.96, 0.99] 0.99 [0.96, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.98, 1.00] 0.93 [0.88, 0.96] 0.00 [0.00, 0.02] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
google/gemini-3.7-flash 2026 0.99 [0.97, 1.00] 0.99 [0.97, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 0.99 [0.96, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.98, 1.00] 0.99 [0.97, 1.00] 1.00 [0.98, 1.00] 0.77 [0.71, 0.82] 1.00 [0.98, 1.00]

Accuracy on parsed replies only

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.98 [0.97, 1.00] 0.98 [0.95, 0.99] 0.98 [0.96, 1.00] 0.98 [0.96, 1.00] 0.97 [0.95, 0.99]
openai/gpt-4o-mini 2024 0.96 [0.94, 0.98] 0.97 [0.94, 0.99] 0.96 [0.94, 0.98] 0.97 [0.94, 0.99] 0.98 [0.96, 0.99]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.94 [0.91, 0.96] 0.95 [0.92, 0.97] 0.95 [0.92, 0.97] 0.93 [0.89, 0.96] 0.94 [0.91, 0.96]
openai/gpt-4.1 2025 0.99 [0.98, 1.00] 0.98 [0.96, 1.00] 0.98 [0.97, 0.99] 0.98 [0.96, 0.99] 0.99 [0.98, 1.00]
openrouter/openai/gpt-5-mini 2025 0.99 [0.98, 1.00] 1.00 [0.99, 1.00] 1.00 [0.99, 1.00] 1.00 [0.99, 1.00] 0.99 [0.99, 1.00]
openrouter/openai/gpt-oss-20b 2025 0.99 [0.97, 1.00] 0.98 [0.97, 1.00] 0.98 [0.96, 0.99] 0.98 [0.97, 1.00] 0.99 [0.97, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [1.00, 1.00] 1.00 [1.00, 1.00] n/a 1.00 [1.00, 1.00] 1.00 [1.00, 1.00]
google/gemini-3.7-flash 2026 0.99 [0.98, 1.00] 0.99 [0.98, 1.00] 1.00 [0.99, 1.00] 1.00 [1.00, 1.00] 0.99 [0.97, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [1.00, 1.00] 1.00 [0.99, 1.00] 1.00 [1.00, 1.00] 1.00 [0.99, 1.00] 1.00 [0.99, 1.00]

Which parser was right when first_anchored and last_anchored disagreed (share right: first / last)

judge vintage reason_then_verdict
openrouter/openai/gpt-oss-20b 2025 first 0.00 [0.00, 0.79] / last 1.00 [0.21, 1.00] (cases=1)

Which parser was right when first_token and last_token disagreed (share right: first / last)

judge vintage reason_then_verdict verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 first 0.08 [0.01, 0.33] / last 0.92 [0.67, 0.99] (cases=13) first 0.00 [0.00, 0.79] / last 1.00 [0.21, 1.00] (cases=1)
openai/gpt-4o-mini 2024 first 0.08 [0.01, 0.33] / last 0.92 [0.67, 0.99] (cases=13)
openrouter/meta-llama/llama-3.1-8b-instruct 2024 first 0.33 [0.06, 0.79] / last 0.67 [0.21, 0.94] (cases=3) first 1.00 [0.21, 1.00] / last 0.00 [0.00, 0.79] (cases=1)
openai/gpt-4.1 2025 first 0.00 [0.00, 0.22] / last 1.00 [0.78, 1.00] (cases=14)
openrouter/openai/gpt-oss-20b 2025 first 0.00 [0.00, 0.49] / last 1.00 [0.51, 1.00] (cases=4)
anthropic/claude-opus-5-5 2026 first 0.00 [0.00, 0.79] / last 1.00 [0.21, 1.00] (cases=1)
openrouter/z-ai/glm-5.3 2026 first 0.00 [0.00, 0.19] / last 1.00 [0.81, 1.00] (cases=16)

Hypothesis evidence

Verdicts on H1 and H2 are written by hand in the findings section below, citing these tables.

H1: 2026 judges, compliance by grader

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
anthropic/claude-opus-5-5 2026 1.00 [0.98, 1.00] 0.93 [0.88, 0.96] 0.00 [0.00, 0.02] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 0.77 [0.71, 0.83] 1.00 [0.98, 1.00]

H1: reasoning judges, preamble rate on G2 and G5

judge vintage verdict_only verdict_then_reason
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
anthropic/claude-opus-5-5 2026 0.07 [0.04, 0.12] 0.00 [0.00, 0.02]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.02] 0.00 [0.00, 0.02]

H2: compliance by judge vintage

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openai/gpt-4o-mini 2024 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.97 [0.94, 0.99] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openai/gpt-4.1 2025 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/openai/gpt-5-mini 2025 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/openai/gpt-oss-20b 2025 0.99 [0.97, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.98, 1.00] 0.93 [0.88, 0.96] 0.00 [0.00, 0.02] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 1.00 [0.98, 1.00] 0.77 [0.71, 0.83] 1.00 [0.98, 1.00]

Findings

To be written after reading the tables above.