Grader comparison: smoke

Dataset 2be79c43878c67bf89ea5c98640d285a7778f3308261850945050bb05e1ee4d2. Git commit fd13b4254478dcf30b944150663730b3aa565ae2. inspect_ai 0.3.263.

Rates are per-case means across epochs, with 95% Wilson intervals over cases; ratios use a bootstrap over cases. Each cell is rate [low, high].

G3 (tool_call) forces the tool call with tool_choice, so its compliance measures the repo’s helper as used, not whether a model chooses to call a tool. Every grader reads verdicts case-insensitively except G3, whose helper validates the grade before upper-casing it; the case tables show how much that costs. Inspect’s OpenAI-compatible provider, which OpenRouter uses, parses a tool call written as text into a real one only under the emulate_tools model arg, which this harness does not set, so G3’s no-tool-call rate has no such rescue.

Slice: adversarial

Compliance

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openrouter/openai/gpt-4o-mini 2024 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 0.00 [0.00, 0.43] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openrouter/openai/gpt-4.1 2025 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openrouter/openai/gpt-5-mini 2025 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openrouter/openai/gpt-oss-20b 2025 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 0.00 [0.00, 0.43] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openrouter/anthropic/claude-opus-5.5 2026 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 0.00 [0.00, 0.43] 0.20 [0.04, 0.62] 1.00 [0.57, 1.00]
openrouter/google/gemini-3.7-flash 2026 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 0.00 [0.00, 0.43] 1.00 [0.57, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]

Sample errors after retries (the failure-mode table splits them into http_4xx, http_5xx, timeout and other)

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-4o-mini 2024 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 1.00 [0.57, 1.00] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-4.1 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/anthropic/claude-opus-5.5 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 1.00 [0.57, 1.00] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/google/gemini-3.7-flash 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]

Truncation

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-4o-mini 2024 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-4.1 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/anthropic/claude-opus-5.5 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/google/gemini-3.7-flash 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]

Case mismatch: verdict readable ignoring case but not in the requested lower case

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.60 [0.23, 0.88]
openrouter/openai/gpt-4o-mini 2024 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] n/a 0.00 [0.00, 0.43] 0.80 [0.38, 0.96]
openrouter/openai/gpt-4.1 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] n/a 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/anthropic/claude-opus-5.5 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] n/a 0.00 [0.00, 0.79] 0.20 [0.04, 0.62]
openrouter/google/gemini-3.7-flash 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] n/a 1.00 [0.57, 1.00]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 1.00 [0.57, 1.00]

G3 failures that were case only (a Yes the helper’s schema rejected)

judge vintage tool_call
openrouter/openai/gpt-3.5-turbo-0613 2023 n/a
openrouter/meta-llama/llama-3.1-8b-instruct 2024 n/a
openrouter/openai/gpt-4o-mini 2024 0.00 [0.00, 0.43]
openrouter/openai/gpt-4.1 2025 n/a
openrouter/openai/gpt-5-mini 2025 n/a
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.43]
openrouter/anthropic/claude-opus-5.5 2026 0.00 [0.00, 0.43]
openrouter/google/gemini-3.7-flash 2026 n/a
openrouter/z-ai/glm-5.3 2026 n/a

Preamble: visible text before the verdict (G2, G5; replies with a verdict token)

judge vintage verdict_only verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-4o-mini 2024 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-4.1 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/anthropic/claude-opus-5.5 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/google/gemini-3.7-flash 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]

Q1: anchored first vs last disagreement (replies where both parse)

judge vintage reason_then_verdict verdict_only verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-4o-mini 2024 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-4.1 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/anthropic/claude-opus-5.5 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/google/gemini-3.7-flash 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/z-ai/glm-5.3 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]

Q1: unanchored token first vs last disagreement (replies where both parse)

judge vintage reason_then_verdict verdict_only verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 0.80 [0.38, 0.96] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 0.40 [0.12, 0.77] 0.00 [0.00, 0.43] 0.20 [0.04, 0.62]
openrouter/openai/gpt-4o-mini 2024 0.40 [0.12, 0.77] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-4.1 2025 0.80 [0.38, 0.96] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-5-mini 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/openai/gpt-oss-20b 2025 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/anthropic/claude-opus-5.5 2026 0.60 [0.23, 0.88] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/google/gemini-3.7-flash 2026 0.00 [0.00, 0.43] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]
openrouter/z-ai/glm-5.3 2026 0.60 [0.23, 0.88] 0.00 [0.00, 0.43] 0.00 [0.00, 0.43]

Epoch agreement on compliance / on the verdict

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a
openrouter/meta-llama/llama-3.1-8b-instruct 2024 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a
openrouter/openai/gpt-4o-mini 2024 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a
openrouter/openai/gpt-4.1 2025 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a
openrouter/openai/gpt-5-mini 2025 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a
openrouter/openai/gpt-oss-20b 2025 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a
openrouter/anthropic/claude-opus-5.5 2026 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a
openrouter/google/gemini-3.7-flash 2026 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a
openrouter/z-ai/glm-5.3 2026 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a 1.00 / n/a

At least one compliant reply in 1 / 2 / 3 attempts

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a
openrouter/meta-llama/llama-3.1-8b-instruct 2024 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a
openrouter/openai/gpt-4o-mini 2024 1.00 / n/a / n/a 1.00 / n/a / n/a 0.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a
openrouter/openai/gpt-4.1 2025 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a
openrouter/openai/gpt-5-mini 2025 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a
openrouter/openai/gpt-oss-20b 2025 1.00 / n/a / n/a 1.00 / n/a / n/a 0.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a
openrouter/anthropic/claude-opus-5.5 2026 1.00 / n/a / n/a 1.00 / n/a / n/a 0.00 / n/a / n/a 0.20 / n/a / n/a 1.00 / n/a / n/a
openrouter/google/gemini-3.7-flash 2026 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 0.00 / n/a / n/a 1.00 / n/a / n/a
openrouter/z-ai/glm-5.3 2026 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a 1.00 / n/a / n/a

Cost per 100 successful grades (USD); mean output / reasoning tokens

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 $0.026; 44 / 0 $0.016; 1 / 0 $0.029; 36 / 0 $0.027; 30 / 0 $0.020; 16 / 0
openrouter/meta-llama/llama-3.1-8b-instruct 2024 $0.001; 45 / 0 $0.000; 2 / 0 $0.002; 38 / 0 $0.005; 31 / 0 $0.001; 15 / 0
openrouter/openai/gpt-4o-mini 2024 $0.006; 51 / 0 $0.002; 1 / 0 n/a; n/a / n/a $0.005; 29 / 0 $0.003; 16 / 0
openrouter/openai/gpt-4.1 2025 $0.072; 47 / 0 $0.033; 2 / 0 $0.074; 39 / 0 $0.059; 32 / 0 $0.046; 17 / 0
openrouter/openai/gpt-5-mini 2025 $0.036; 157 / 77 $0.014; 50 / 81 $0.075; 348 / 256 $0.023; 96 / 67 $0.021; 87 / 67
openrouter/openai/gpt-oss-20b 2025 $0.002; 76 / 53 $0.001; 50 / 45 $0.000; 42 / 0 $0.002; 37 / 13 $0.003; 70 / 49
openrouter/anthropic/claude-opus-5.5 2026 $0.286; 92 / 27 $0.103; 4 / 0 n/a; n/a / n/a $0.203; 51 / 0 $0.162; 32 / 1
openrouter/google/gemini-3.7-flash 2026 $0.042; 77 / 53 $0.033; 58 / 57 $0.050; 95 / 57 $0.046; 91 / 51 $0.037; 66 / 46
openrouter/z-ai/glm-5.3 2026 $0.074; 118 / 65 $0.033; 34 / 34 $0.096; 90 / 51 $0.058; 82 / 51 $0.051; 65 / 48

Failure modes (top three)

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 none none none none none
openrouter/meta-llama/llama-3.1-8b-instruct 2024 none none none none none
openrouter/openai/gpt-4o-mini 2024 none none sample_error:http_4xx (5) none none
openrouter/openai/gpt-4.1 2025 none none none none none
openrouter/openai/gpt-5-mini 2025 none none none none none
openrouter/openai/gpt-oss-20b 2025 none none grader_no_tool_call (5) none none
openrouter/anthropic/claude-opus-5.5 2026 none none sample_error:http_4xx (5) grader_parse_failure (4) none
openrouter/google/gemini-3.7-flash 2026 none none none grader_parse_failure (5) none
openrouter/z-ai/glm-5.3 2026 none none none none none

Adversarial tag seed: agreement with the hand label

judge vintage reason_then_verdict verdict_only tool_call structured_output verdict_then_reason
openrouter/openai/gpt-3.5-turbo-0613 2023 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openrouter/meta-llama/llama-3.1-8b-instruct 2024 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 0.80 [0.38, 0.96] 1.00 [0.57, 1.00]
openrouter/openai/gpt-4o-mini 2024 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 0.00 [0.00, 0.43] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openrouter/openai/gpt-4.1 2025 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openrouter/openai/gpt-5-mini 2025 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openrouter/openai/gpt-oss-20b 2025 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 0.00 [0.00, 0.43] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]
openrouter/anthropic/claude-opus-5.5 2026 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 0.00 [0.00, 0.43] 0.20 [0.04, 0.62] 1.00 [0.57, 1.00]
openrouter/google/gemini-3.7-flash 2026 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 0.00 [0.00, 0.43] 1.00 [0.57, 1.00]
openrouter/z-ai/glm-5.3 2026 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00] 1.00 [0.57, 1.00]

Hypothesis evidence

Verdicts on H1 and H2 are written by hand in the findings section below, citing these tables.

H1: 2026 judges, compliance by grader

No replies.

H1: reasoning judges, preamble rate on G2 and G5

No replies.

H2: compliance by judge vintage

No replies.

Findings

To be written after reading the tables above.