Empty responses: gate 2 request

October 23, 2026

Gate 2 request

Outcome (2026-10-23). Matt Fisher approved A1, A2, A7 and A8 and the budget, and chose A9 option (a), with AIR-Bench dropped from the three failing routes. The freeze moved from 4038b92 to 4a61367, after main (PR #10’s Wilson clamp) was merged into the branch; the smoke tables recomputed there are identical. Recorded in experiment.yaml.

The build is smoke-tested, and nothing from the main run has started. This note asks for gate 2: approval of the analysis code to freeze, the amendments, and the budget for the main run, re-runs and Part C. It covers plan.md steps 3-7: exploration, the smoke runs, analysis on the smoke data, the cost re-estimate, and this request.

Spent so far: $4.41, all smoke: $3.66 Part A, $0.16 Part C, $0.60 re-runs. A further $0.05 or less went on one-line route checks made outside the runner. The ledger lists 173 run records and 3 analyses.

What I need from you

  1. Approve A1 and A2 (proposed after the build review) and A7 and A8 (from the smoke run). A3-A6 are recorded below; you approved them in the session.
  2. Decide A9, AIR-Bench on models that reject a trailing assistant turn. This is the one finding that touches a decision rule, so I have not acted on it.
  3. Approve the budget, unchanged: main $120, re-runs $25, Part C $25, within the $200 ceiling.
  4. Freeze the analysis code at commit 4038b92.

Environment check

Every key, host, bucket read and price check passed. Three models could not be used as specified:

Problem Change Amendment
openai/gpt-5-mini (and gpt-5) needs OpenAI organisation verification OpenAI reasoning route is openai/gpt-5.4-mini A4 (you asked for this)
Google refuses gemini-2.5-pro for this key, so FORTRESS’s panel could not call it panel calls openrouter/google/gemini-2.5-pro A5 (you chose this)
Google serves gemini-3.7-flash as gemini-3.8-flash both Gemini routes name gemini-3.8-flash A6 (you chose this)
Claude 3.7 Sonnet, FORTRESS’s Anthropic judge, is retired anthropic/claude-sonnet-5 A3 (approved before this)

Exploration

From exploration.md. I scanned grader-parsing’s 50 answer-generation logs from the bucket: 49,600 SimpleQA Verified replies from 43 models. 99.8% are answers. The non-answers are 54 errors, 34 silent, 7 filtered, 3 truncated_empty and 3 truncated_text. Every form the classifier separates occurs in real logs:

  • GPT-5-family filter refusals with no text on benign factual questions
  • Alibaba refusing through HTTP 400 “inappropriate content” errors
  • Gemini 3 Pro reasoning and then writing nothing
  • Claude Opus 4.6 using all 32,000 tokens with no text

The Hawk imports carry no model events, so for exploration only they were classified from sample.output. The experiment’s own logs all have events.

The smoke runs

Config What ran Spent Cap
smoke.yaml 11 evals × their routes × 5 samples, plus HLE tight on the 4 reasoning routes $3.66 $8
smoke_part_c.yaml 28 judge × non-answer form cells, 2 samples, 1 repeat $0.16 $2
smoke_reruns.yaml wide (2) and control (23) re-runs of the smoke’s own picks $0.60 $1

smoke.yaml spent $3.66 in total: $3.18 on the first pass, $0.44 re-running the seven Sonnet-via-OpenRouter steps under A7, and $0.03 when eval_set retried one cell’s errored samples on resume. plan.md had no smoke for the re-run arms, so I added smoke_reruns.yaml (a ruling) to put every arm through a paid run, as the framework asks.

The tables are in results/smoke/summary.md, produced by summarise at 9fba4db. With 5 samples a cell, they show that the pipeline works, not what the rates are.

Checks

  • P0 check: matches in 60 of 62 cells. The two others are AIR-Bench on Gemini direct and on Sonnet via OpenRouter. Every sample in both errored (A9), so neither the log nor the rebuild has a headline. No cell disagrees.
  • Each eval’s own reply was extracted. All 330 replies came from the sample’s first model event. On AIR-Bench, the extracted reply is never shaped like the judge’s verdict, and the final output always is, so the reply is the first generation, not the judge’s.
  • Settings reached the provider. I read one logged request per route:
    • gpt-5.4-mini: reasoning: {effort: medium}, max_output_tokens.
    • Sonnet direct: output_config: {effort: medium}, adaptive thinking.
    • Sonnet via OpenRouter: reasoning: {effort: medium}, pinned to Anthropic under A7.
    • Gemini direct: BLOCK_NONE on all five categories, includeThoughts.
    • max_tokens is 256 on XSTest and CoCoNot and 2,048 on StrongREJECT for every route. HLE sends none, except Sonnet direct, which sends 42,000 (32,000 plus Inspect’s effort bonus).
  • Errored samples are counted. The logs hold 13 sample errors, and the table has 13 rows of class error.
  • Ledger. Every run record has a non-null spent_usd and an empty unpriced_models.

What the smoke found

  1. AIR-Bench fails on Sonnet 5 and on Gemini 3.8 Flash direct (A9). AIR-Bench’s annotate solver puts the judge prompt in the user message but leaves the model’s reply as the last message. Both Sonnet routes reject that request (“This model does not support assistant message prefill”), and so does Gemini direct (“Requests ending with a model turn are not supported”). So every AIR-Bench sample where the model answered errors on those three routes. Only replies that skip the judge are scored: empty text, or Sonnet’s filtered replies. Gemini direct’s “BLOCKED” text is not empty, so it passes AIR-Bench’s guard, goes to the judge, and errors too. Anyone running AIR-Bench at the pinned commit on these models hits this, so it is a finding for the report. It also leaves three of the six AIR-Bench D3 primary cells without a headline.
  2. Sonnet 5 via OpenRouter could not run evals that set temperature (A7). No OpenRouter endpoint for Sonnet 5 accepts temperature, so with require_parameters every XSTest, CoCoNot and StrongREJECT request returned 404. These are kept in the superseded logs as class error. Pinned to Anthropic, OpenRouter drops the parameter, as the direct adapter does.
  3. The classifier was reading attachment references (fixed, 0bb669b). Inspect stores long message text in the log as attachment://<hash>, so load_replies saw 45 characters instead of the reply. That would have let long provider text (Gemini’s “BLOCKED” with its safety ratings) pass as an answer. Logs are now read with attachments resolved, and a test covers it.
  4. At 256 tokens, most XSTest and CoCoNot replies are cut off with text. They are truncated_text: 29 of 30 on XSTest safe, 16 of 30 on XSTest unsafe, 20 of 30 on CoCoNot. Few stop with no text: one truncated_empty on each of XSTest safe and CoCoNot, both gpt-5.4-mini. D1(a) counts only truncated_empty, so it may stay low on these evals while the judge grades replies that were cut off. This does not change the question or a decision rule. A2 already treats a truncated_text refusal by the scorer’s verdict, and the report will show the truncated_text share next to D1(a).
  5. Truncation on HLE. At the tight 2,048-token limit, 4 of 5 gpt-5.4-mini replies and 2 of 5 Sonnet replies were truncated_empty. At its own settings Sonnet direct ran one reply to 42,000 tokens with no text. One such reply costs about $0.42, which is why HLE dominates the cost estimate.
  6. Gemini filters capability questions. 1 of 5 HLE replies on Gemini direct and 2 of 5 on OpenRouter came back content_filter. Gemini also filtered 4 of 5 FORTRESS adversarial and AIR-Bench prompts on OpenRouter (Inspect’s direct route runs at BLOCK_NONE).
  7. D2’s two readings split, as A1 anticipated. Under either, 3 of 16 Sonnet samples and 5 of 18 Gemini samples disagree on form (19% and 28%). Under both, 0 of 13 for each. On the main data, A1’s rule applies if the readings fall on opposite sides of 10%.
  8. The CoCoNot judge passes every non-answer it saw. 2 of 2 for each of the six forms. StrongREJECT, XSTest, HLE, FORTRESS benign and the FORTRESS panel passed none, apart from one “(no content)” on FORTRESS benign. At n = 2 this is a signal only.
  9. Runner and ledger fixes. Run records now name the commit each step ran from. On resume, they had repeated the run’s starting commit. The A7 lines written before the fix say d985b6c, and the Inspect logs record the true revision. summarise now appends an analysis record, with its dirty flag checked before it writes output, and per-reply tables are kept out of git because their error text can quote prompts.

Nothing here changes the question or the hypotheses. Finding 1 changes which D3 primary cells can be measured, which is why A9 is a question.

Cost re-estimate

Each smoke step’s cost per sample, scaled to the main run’s sample counts:

Config Estimate Cap Main driver
main $91 (bootstrap 90% range $75-113; P(> cap) ≈ 1%) $120 HLE $30 (Sonnet direct alone $13), FORTRESS adversarial $22, XSTest $18
re-runs $2-6 $25 about 80 wide re-runs at 16,000 tokens, plus 50 control; an HLE Sonnet pick costs about $0.42
Part C $8 $25 FORTRESS panel $2.9, StrongREJECT $2.3, XSTest $2.2
spent $4.41 $11 smoke
total about $105-130 ceiling $200

The main estimate per route: Sonnet direct $29, gpt-5.4-mini $17, gpt-4.1 $14, Gemini direct $12, Sonnet OpenRouter $10, Gemini OpenRouter $9. Part B makes no new judge calls, because P3 never judges and P2 leaves samples unscored, so the spec’s $5 for it is unused. If A9 option (a) is chosen, dropping AIR-Bench from the three failing routes saves about $4. No cut is needed. The spec’s list of cuts (FORTRESS adversarial to 75 and so on) stays in reserve, and the runner’s per-config caps stop a run before the ceiling.

Sample sizes (spec section 7)

The spec’s sizes stand, and I propose no change. On the smoke data, the evals that provoke non-answers do so as planned:

Eval Non-answers, default arm, all routes
FORTRESS adversarial 30%
AIR-Bench 23% (mostly errors, per A9)
HLE 20%
CoCoNot 3%
XSTest safe 3%

On XSTest and CoCoNot, the truncations concentrate on gpt-5.4-mini (1 of 5 in each). For D1(a), a 10% rate on XSTest safe (250 samples) has a 95% interval of about ±4 points, and 3% about ±2, as the spec assumed.

Projected to the main run:

  • D1(b) gets roughly 80 truncations to re-run, mostly gpt-5.4-mini on XSTest and CoCoNot, enough for a pooled rate of ±11 points or better.
  • D2 has a refusal by either route on about half the paired samples. That is several hundred samples per pair, giving ±4-5 points.

One consequence to expect: on the XSTest D3 primary cells, non-answers are about 3% of samples, while the threshold is the P0 interval’s half-width (about 5-6 points at 250 samples). So XSTest is unlikely to meet D3. That is a result, not a reason to change the size.

Amendments

amendments.md has the full text.

  • A1, for approval. D2 is reported under two readings, either (the decision rule’s) and both. If they fall on opposite sides of 10%, D2 is not treated as decided.
  • A2, for approval. A text refusal is the eval’s own refusal verdict. FORTRESS adversarial is left out of D2.
  • A3, for the record. FORTRESS’s Anthropic judge is Claude Sonnet 5. Approved before the smoke.
  • A4-A6, for the record. GPT-5.4 mini replaces gpt-5-mini, FORTRESS’s Gemini judge is called through OpenRouter, and both Gemini routes use 3.8 Flash. Approved in the session.
  • A7, for approval. Sonnet 5 via OpenRouter is pinned to Anthropic’s endpoint, not run with require_parameters.
  • A8, for approval. wide re-runs only truncations whose sent limit was under 16,000 tokens. A truncation at Sonnet’s 42,000 default cannot be widened at 16,000, and D1(b)’s refusal evals all run at 2,048 or less.
  • A9, your decision. AIR-Bench on the three routes that reject a trailing assistant turn. Options:
    • (a) Run AIR-Bench as is (recommended). Report the failure as a finding, and take the D3 AIR-Bench primary cells from the three routes where it runs: gpt-4.1, gpt-5.4-mini and Gemini via OpenRouter. This keeps the eval exact, and it matches what anyone running it today gets. It loses the AIR-Bench cells for the Sonnet pair and Gemini direct, and AIR-Bench’s D2 contribution for both pairs. Optionally, drop AIR-Bench from those three routes in the main run to save about $4, since their samples can only error.
    • (b) Patch annotate locally. Drop the trailing assistant message before the judge call, which is probably the upstream fix. That restores all six cells, but the eval is no longer exact on any route, and the report must say so. It needs a code change in tasks.py, a test, and an AIR-Bench re-smoke on the three routes (about $0.20).
    • (c) Remove AIR-Bench from D3’s primary cells. D3 then rests on XSTest and FORTRESS benign, and AIR-Bench becomes exploratory.

Analysis code to freeze

Commit 4038b92. It holds the analysis as run on the smoke data, in four files:

  • src/eval_experiments/empty_responses/extract.py
  • src/eval_experiments/empty_responses/classify.py
  • src/eval_experiments/empty_responses/policies.py
  • src/eval_experiments/empty_responses/analyse.py

The smoke tables were computed with it (at 9fba4db, which changed only data and docs). After gate 2, any change to these files is an amendment, and the pre-registered numbers are computed at this commit. If A9 (b) is chosen, the patch goes in tasks.py, which is task code, not analysis, so the freeze is unaffected.

At 4038b92 442 tests pass, basedpyright reports no errors, and pre-commit is clean.