Rulings

Every decision taken on Matt’s behalf during the build and runs, in order, each with what it costs if wrong.

  • Adapter-written text counts as no text. The spec’s class table puts error_text before truncated_empty, but does not say what happens when an adapter writes error text and stops for length (Anthropic’s “prompt is too long” sets ModelOutput.error, the error text and model_length). The classifier treats any adapter-written text (ModelOutput.error set, or a known provider form) as no text. It is error_text unless the stop reason is max_tokens or model_length, in which case the rules below it apply, so a context overflow is truncated_empty, not a refusal. Cost if wrong: a few overflow replies counted as truncations rather than error text; both are non-answers, and D1 reports them.
  • Google forms only in error_text patterns. OpenAI policy errors, Grok safety errors and Google prompt blocks all arrive with stop_reason="content_filter", so they are classed filtered before any pattern is checked. The patterns therefore cover only Google’s "(no content)" (stop reason stop) and "BLOCKED: " text. Cost if wrong: none for the class counts; D2 compares forms using filtered and error_text together as provider-signalled refusals.
  • The reply is the model’s first generation. AIR-Bench’s annotate solver grades the reply with the model under test itself (notes/eval-scoring.md), so “the model’s last call before scoring” would read the judge’s verdict as the reply. Every eval here answers in one generation, so extraction takes the model’s first call outside any scorer, as the spec’s “first assistant message after generation” says. Cost if wrong: none for these evals; a multi-turn eval added later would need its own rule.
  • AIR-Bench’s judge is the model under test. The spec’s Part A table says its judge runs inside the solver, but not that it is the same model. So AIR-Bench’s judge differs by route, and its judge calls are billed to the model under test. The report states this as a fidelity note. Cost if wrong: none; it is how the eval runs.
  • BFCL simple is simple_python. The spec says “BFCL simple”; the eval has no category of that name, and simple_python is the single-call Python category. Cost if wrong: none.
  • GPQA Diamond runs one epoch. Its task defaults to 4 epochs; the spec says every route runs each eval once, so the runner’s epochs: 1 applies. Cost if wrong: wider intervals on GPQA, which is not a primary cell.
  • Headlines are rebuilt from per-sample contributions. Part B does not re-run each eval’s metric code. It recomputes each eval’s headline (named in notes/eval-scoring.md) as the mean of per-sample contributions, x 100. Every eval runs one epoch, so a logged score is what the metric sees. The P0 values are compared with the logs’ own reported headline as a check before any policy result is used. Cost if wrong: a P0 that does not match the log, which that check catches.
  • “Failure” is read literally per eval, including where it is odd. P1 and P3 score some non-answers as the eval’s failure value. On XSTest unsafe and FORTRESS adversarial that means “complied” or “full harm”, and on BFCL irrelevance a provider refusal under P3 becomes NOANSWER (0), although abstaining is correct there. The policies are pre-registered, so they are applied as written, and the report points out these cells. Cost if wrong: misleading cells in exploratory tables; none of them is a D3 primary cell except FORTRESS benign, where failure (refusal) is the natural reading.
  • Judges are priced from token usage. The Inspect Costs Plugin prices only the model under test, so the runner prices each judge from its token counts using the same llm-prices API, and stops a paid run if any model cannot be priced. Cost if wrong: list prices, not billed ones, for direct-provider judges; OpenRouter’s billed cost is still recorded per call in the logs.
  • Sample errors do not fail a step. The runner passes fail_on_error=False, so an errored sample stays in the log as class error instead of failing the eval × route cell. Cost if wrong: none; the spec counts errors.
  • Part C runs replace the eval’s epoch reducer. Passing epochs to Inspect drops a task’s own reducer (CoCoNot’s, HLE’s). D4 reads each repeat’s raw score, so it is unaffected, but the Part C logs’ own metrics are not the eval’s, and the P0 check is not run on them. Cost if wrong: none for D4.
  • Part C’s Google block text is what the adapter writes. "BLOCKED: BlockedReason.SAFETY", the string form of the SDK’s enum, not the spec’s "BLOCKED: SAFETY"; the real text also carries the block message and safety ratings on further lines, which Part C leaves out. Cost if wrong: a judge may react to the longer real text differently; noted in the report.
  • The recorded max_tokens is what the provider received. It is read from the logged request (max_tokens, max_output_tokens, max_completion_tokens or maxOutputTokens), falling back to Inspect’s config. Cost if wrong: none.
  • P3 leaves tool_only to the eval’s scorer. The spec’s P3 does not name tool_only. With tools offered it is an answer and not re-routed; without them (no eval here) P3 keeps the eval’s own score. Cost if wrong: none for these evals.
  • D3’s interval is a normal approximation. The half-width that sets D3’s threshold is 1.96 standard errors of the mean contribution, not a Wilson interval, because contributions are not all 0/1 (FORTRESS adversarial, AIR-Bench). Cost if wrong: a slightly different threshold where it exceeds the 1-point floor.
  • Google’s malformed-function-call apology is not classed as adapter text. The adapter writes an apology with stop reason unknown; only BFCL can produce it, and it classes as answer. Cost if wrong: a few BFCL replies classed as answers; BFCL is exploratory.
  • The anthropic package is raised to 2.x. Inspect 0.3.283 refuses to call Anthropic with anthropic below 2.0.0, but pyproject.toml asked only for >=1.0.0 and the lockfile held 1.15.0, so every Anthropic call would have failed. The requirement is now >=2.0.0 (2.1.0 locked), approved by Matt on 2026-10-23; no other package moved. Earlier experiments reproduce at their own commits. Cost if wrong: none for this experiment; grader-parsing re-run at this commit would use the newer SDK.
  • Sonnet 5 judges FORTRESS at its provider defaults. A3 replaces the retired judge with anthropic/claude-sonnet-5. A judge does not inherit the task’s config (notes/eval-scoring.md), so it runs without the routes’ reasoning_effort: medium, as the retired judge did. Cost if wrong: a judge that might grade differently with thinking on; it is one vote of three.
  • The Anthropic key is read from EVAL_EXPERIMENTS_ANTHROPIC_API_KEY in cloud sessions. Claude Code cloud sessions strip ANTHROPIC_API_KEY from the environment, because the session itself authenticates with Anthropic. The runner’s load_env() copies EVAL_EXPERIMENTS_ANTHROPIC_API_KEY into ANTHROPIC_API_KEY for its own process when the latter is unset, and leaves a set key alone; approved by Matt on 2026-10-23. The session also sets ANTHROPIC_BASE_URL, which Inspect reads; it points at api.anthropic.com, so calls still reach the public API. Cost if wrong: none for results; a run on another machine uses ANTHROPIC_API_KEY as before.
  • GPT-5.4 mini replaces gpt-5-mini (A4). Matt asked for an OpenAI model that needs no organisation verification. Of the models the account can call on the Responses API (gpt-5-nano, gpt-5.1, gpt-5.4-mini, gpt-5.4-nano, o3, o3-mini, o4-mini), GPT-5.4 mini is the closest to the spec’s role: a GPT-5-family mini that reasons at effort medium, so reasoning competes with the visible reply for max_output_tokens. gpt-5.4-nano used no reasoning tokens on a test call at medium, and gpt-5-nano is a much weaker model. Cost if wrong: about 2-3 times gpt-5-mini’s price on this route, and truncation rates that describe a newer mini rather than gpt-5-mini; the report names the model as run.
  • Sonnet 5 via OpenRouter is pinned to Anthropic, not run with require_parameters (A7, proposed for gate 2). require_parameters failed every sample of the evals that set temperature. Pinning the route to Anthropic’s endpoint keeps the comparison with the direct route to route and adapter alone. The smoke steps for this route were re-run under A7, at about $0.3, within the smoke cap; the first attempt’s logs are kept under data/empty-responses/logs/smoke_superseded/. Cost if wrong: if A7 is rejected, the route goes back to require_parameters and these evals have no OpenRouter Sonnet data, losing four of the seven refusal evals from the Sonnet D2 pair.
  • Per-reply tables are not committed. summarise writes replies.csv, whose sample_error column holds provider error messages, and some of those echo the full request, benchmark prompt included. .gitignore now excludes experiments/*/results/**/replies.csv; the tables go to the bucket with the logs, as in grader-parsing. Cost if wrong: none; every committed table is an aggregate.
  • Long replies are read through their attachments. Inspect stores long message text in a log as attachment://<hash>, and load_replies read samples without resolving it, so the classifier saw a 45-character reference instead of the text. Fixed before the freeze, with a test (commit 0bb669b). Class counts on the smoke data changed only for text_chars, since no long reply matched a provider pattern. Cost if wrong: none.
  • Each ledger record names the commit its step ran from. The runner wrote run.json’s starting commit into every record, so a resumed run (the A7 smoke re-run) recorded the first run’s commit, d985b6c, for steps run at 039915e. Records now carry the current commit, and run.json keeps the starting one. The smoke ledger’s A7 lines were written before the fix and say d985b6c; the Inspect logs record the true revision. Cost if wrong: none.
  • summarise appends an analysis record to the ledger. The framework requires a ledger line for every analysis computed, and summarise wrote none. It now appends one with its inputs, output directory, commit and dirty flag. Cost if wrong: none.
  • A resumed eval_set retries errored samples. Re-running smoke.yaml for A7 also re-ran AIR-Bench on Gemini direct, whose five samples had all errored ($0.03; the errors recurred). In the main run, a resume would retry errored samples, so an error rate can depend on whether a step was resumed. The main run is started once. If it must be resumed, the report says which steps were, and each step’s ledger records show it. Cost if wrong: error-class rates from a resumed step would be lower than a single pass gives.
  • The wide and control arms were smoke-tested with a small re-run config. plan.md’s smoke covered only default, tight and Part C, but the framework’s smoke stage requires a run through every arm. make_configs now also writes configs/smoke_reruns.yaml (cap $1), which re-runs the ids select-reruns picks from the smoke data, under A8. Cost if wrong: under $1 of the smoke cap.
  • The analysis is frozen after merging main. PR #10, merged into main on 2026-10-23, clamps grader_comparison.stats.wilson to [0, 1], and the empty-responses analysis imports it. So that the main run and the pre-registered numbers use the same code, the freeze is at the commit after main was merged into this branch, not at 4038b92. The smoke tables were recomputed there and compared with the committed ones. Cost if wrong: none; the clamp moves a bound only by rounding error.