Grader comparison: smoke2

Dataset 2be79c43878c67bf89ea5c98640d285a7778f3308261850945050bb05e1ee4d2. Git commit 87f78a25b5be147bda49e5093325d92ce4e887d4. inspect_ai 0.3.263.

Rates are per-case means across epochs, with 95% Wilson intervals over cases; ratios use a bootstrap over cases. Each cell is rate [low, high].

G3 (tool_call) forces the tool call with tool_choice, so its compliance measures the repo’s helper as used, not whether a model chooses to call a tool. Every grader reads verdicts case-insensitively except G3, whose helper validates the grade before upper-casing it; the case tables show how much that costs. Inspect’s OpenAI-compatible provider, which OpenRouter uses, parses a tool call written as text into a real one only under the emulate_tools model arg, which this harness does not set, so G3’s no-tool-call rate has no such rescue.

Slice: adversarial

Compliance

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openai/gpt-4o-mini 2024 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openai/gpt-4.1 2025 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openrouter/openai/gpt-5-mini 2025 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openrouter/openai/gpt-oss-20b 2025 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 0.00 [0.00, 0.43] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]

Sample errors after retries (the failure-mode table splits them into http_4xx, http_5xx, timeout and other)

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openai/gpt-4.1 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 1.00 [0.57, 1.00] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]

Truncation

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openai/gpt-4.1 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]

Case mismatch: verdict readable ignoring case but not in the requested lower case

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.80 [0.38, 0.96]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.40 [0.12, 0.77]
openai/gpt-4.1 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.20 [0.04, 0.62]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] n/a 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 1.00 [0.57, 1.00]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.80 [0.38, 0.96]

G3 failures that were case only (a Yes the helper’s schema rejected)

judge vintage tool_call
openrouter/openai/gpt-3.5-turbo-0613 2023 n/a
openai/gpt-4o-mini 2024 n/a
openrouter/meta-llama/llama-3.1-8b-instruct 2024 n/a
openai/gpt-4.1 2025 n/a
openrouter/openai/gpt-5-mini 2025 n/a
openrouter/openai/gpt-oss-20b 2025 n/a
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.43]
google/gemini-3.7-flash 2026 n/a
openrouter/z-ai/glm-5.3 2026 n/a

Preamble: visible text before the verdict (G2, G5; replies with a verdict token)

judge vintage verdict_only verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openai/gpt-4.1 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]

Q1: anchored first vs last disagreement (replies where both parse)

judge vintage reason_then_verdict verdict_only verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openai/gpt-4o-mini 2024 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openai/gpt-4.1 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
anthropic/claude-opus-5-5 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]

Q1: unanchored token first vs last disagreement (replies where both parse)

judge vintage reason_then_verdict verdict_only verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.60 [0.23, 0.88] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openai/gpt-4o-mini 2024 0.40 [0.12, 0.77] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openai/gpt-4.1 2025 0.80 [0.38, 0.96] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-oss-20b 2025 0.40 [0.12, 0.77] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
anthropic/claude-opus-5-5 2026 0.20 [0.04, 0.62] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
google/gemini-3.7-flash 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/z-ai/glm-5.3 2026 0.60 [0.23, 0.88] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]

Epoch agreement on compliance / on the verdict

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a
openai/gpt-4o-mini 2024 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a
openrouter/meta-llama/llama-3.1-8b-instruct 2024 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a
openai/gpt-4.1 2025 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a
openrouter/openai/gpt-5-mini 2025 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a
openrouter/openai/gpt-oss-20b 2025 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a
anthropic/claude-opus-5-5 2026 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a
google/gemini-3.7-flash 2026 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a
openrouter/z-ai/glm-5.3 2026 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a

At least one compliant reply in 1 / 2 / 3 attempts

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a
openai/gpt-4o-mini 2024 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a
openrouter/meta-llama/llama-3.1-8b-instruct 2024 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a
openai/gpt-4.1 2025 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a
openrouter/openai/gpt-5-mini 2025 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a
openrouter/openai/gpt-oss-20b 2025 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a
anthropic/claude-opus-5-5 2026 1.00 / n/a / n/a 1.00 / n/a / n/a 0.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a
google/gemini-3.7-flash 2026 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a
openrouter/z-ai/glm-5.3 2026 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a

Cost per 100 successful grades (USD); mean output / reasoning tokens

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 $0.026; 46 / 0 $0.016; 1 / 0 $0.030; 39 / 0 $0.027; 29 / 0 $0.020; 18 / 0
openai/gpt-4o-mini 2024 $0.006; 50 / 0 $0.002; 1 / 0 $0.005; 36 / 0 $0.005; 28 / 0 $0.003; 17 / 0
openrouter/meta-llama/llama-3.1-8b-instruct 2024 $0.001; 43 / 0 $0.000; 2 / 0 $0.002; 35 / 0 $0.005; 31 / 0 $0.000; 17 / 0
openai/gpt-4.1 2025 $0.069; 43 / 0 $0.033; 1 / 0 $0.070; 33 / 0 $0.065; 28 / 0 $0.046; 16 / 0
openrouter/openai/gpt-5-mini 2025 $0.028; 120 / 64 $0.014; 49 / 34 $0.076; 355 / 282 $0.022; 90 / 44 $0.018; 72 / 43
openrouter/openai/gpt-oss-20b 2025 $0.003; 108 / 72 $0.002; 92 / 96 $0.003; 101 / 68 $0.002; 87 / 60 $0.003; 81 / 66
anthropic/claude-opus-5-5 2026 $0.211; 54 / 12 $0.103; 4 / n/a n/a; n/a / n/a $0.300; 43 / n/a $0.156; 29 / n/a
google/gemini-3.7-flash 2026 $0.042; 78 / 54 $0.031; 51 / 50 $0.057; 95 / 55 $0.042; 80 / 51 $0.042; 79 / 60
openrouter/z-ai/glm-5.3 2026 $0.059; 104 / 54 $0.037; 41 / 45 $0.083; 99 / 53 $0.043; 76 / 45 $0.047; 59 / 44

Failure modes (top three)

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 none none none none none
openai/gpt-4o-mini 2024 none none none none none
openrouter/meta-llama/llama-3.1-8b-instruct 2024 none none none none none
openai/gpt-4.1 2025 none none none none none
openrouter/openai/gpt-5-mini 2025 none none none none none
openrouter/openai/gpt-oss-20b 2025 none none none none none
anthropic/claude-opus-5-5 2026 none none sample_error:http_4xx (5) none none
google/gemini-3.7-flash 2026 none none none none none
openrouter/z-ai/glm-5.3 2026 none none none none none

Adversarial tag seed: agreement with the hand label

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openai/gpt-4o-mini 2024 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openai/gpt-4.1 2025 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openrouter/openai/gpt-5-mini 2025 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openrouter/openai/gpt-oss-20b 2025 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
anthropic/claude-opus-5-5 2026 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 0.00 [0.00, 0.43] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
google/gemini-3.7-flash 2026 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]

Hypothesis evidence

Verdicts on H1 and H2 are written by hand in the findings section below, citing these tables.

H1: 2026 judges, compliance by grader

No replies.

H1: reasoning judges, preamble rate on G2 and G5

No replies.

H2: compliance by judge vintage

No replies.

Findings

To be written after reading the tables above.