Smoke-run cost estimate for phase one (smoke2)
From 225 smoke calls (5 seed cases x 5 graders x 9 judges, one epoch). Routing: gpt-4o-mini, gpt-4.1 (OpenAI), Opus 5.5 (Anthropic) and Gemini 3.7 Flash (Google) direct; the rest via OpenRouter with require_parameters (gpt-oss-20b also ignores CoreWeave). OpenRouter judges use billed usage.cost; direct judges use Inspect total_cost from llm-prices. Projection: mean cost per call x 6,000 generic calls + mean x 1.5 x 2,400 repo-scorer calls.
| judge | cost per call (USD) | mean output tokens | mean reasoning tokens | projected (USD) |
|---|---|---|---|---|
| claude-opus-5-5 | 0.00192 | 32 | 12 | 18.46 |
| gpt-4.1 | 0.00056 | 24 | 0 | 5.42 |
| glm-5.3 | 0.00054 | 76 | 48 | 5.18 |
| gemini-3.7-flash | 0.00043 | 76 | 54 | 4.10 |
| gpt-5-mini | 0.00032 | 137 | 93 | 3.05 |
| gpt-3.5-turbo-0613 | 0.00024 | 27 | 0 | 2.29 |
| gpt-4o-mini | 0.00004 | 26 | 0 | 0.42 |
| gpt-oss-20b | 0.00002 | 94 | 72 | 0.23 |
| llama-3.1-8b-instruct | 0.00002 | 25 | 0 | 0.16 |
Total projected phase one: $39.31. Smoke2 cost $0.0928.
Seed cases are short; real SimpleQA answers will raise input tokens, so allow 2x.