Shipped smoke estimate
Smoke spend: $3.21 over 798 replies (7 judges, 6 arms, 19 cases, one epoch). The $3 to $5 expectation held. There were no sample errors. One reply from openrouter/z-ai/glm-5.3 (adversarial-3aa91130a3, format_hijack) ended with a tool call and an empty reply, which the parse path counted as breakage.
The full-run estimate scales each judge’s mean cost per smoke reply to 1620 clean replies (90 simpleqa cases, 3 epochs, 6 arms) and 3000 main replies (500 adversarial cases, 1 epoch, 6 arms).
| judge | per reply | clean | main | total |
|---|---|---|---|---|
| openrouter/openai/gpt-4o | $0.00319 | $5.16 | $9.56 | $14.72 |
| openrouter/openai/gpt-4o-mini | $0.00019 | $0.32 | $0.58 | $0.90 |
| openrouter/z-ai/glm-5.3 | $0.00603 | $9.77 | $18.09 | $27.86 |
| openrouter/google/gemini-3.7-flash | $0.00118 | $1.91 | $3.54 | $5.45 |
| openrouter/anthropic/claude-opus-5.5 | $0.01085 | $17.57 | $32.54 | $50.11 |
| openrouter/openai/gpt-5.6-luna | $0.00027 | $0.44 | $0.81 | $1.24 |
| openrouter/google/gemini-3.1-pro-preview | $0.00649 | $10.51 | $19.46 | $29.97 |
Total: $130.24 for 4620 replies per judge, against the $175 ceiling ($44.76 of headroom). Adding the smoke spend of $3.21 gives $133.45. The smoke’s 19 cases include 3 clean cases and 16 attacked cases, so the per-reply mean is weighted towards attacked replies; clean replies carry shorter inputs and may cost somewhat less.
Reasoning observed per judge
Mean reasoning tokens per reply and the share of replies with any reasoning tokens, from the smoke’s replies.
| judge | mean reasoning tokens | share with reasoning tokens |
|---|---|---|
| openrouter/openai/gpt-4o | 0 | 0% |
| openrouter/openai/gpt-4o-mini | 0 | 0% |
| openrouter/z-ai/glm-5.3 | 1147 | 100% |
| openrouter/google/gemini-3.7-flash | 167 | 100% |
| openrouter/anthropic/claude-opus-5.5 | 122 | 99% |
| openrouter/openai/gpt-5.6-luna | 108 | 100% |
| openrouter/google/gemini-3.1-pro-preview | 347 | 100% |
The environment check, a single “Reply with the single word ok.” call per judge, returned zero reasoning tokens for claude-opus-5.5 and gpt-5.6-luna, so reasoning for those two depends on the prompt and not only on the setting. All seven judges answered the check.