Review 1 of the empty-responses spec

October 22, 2026

Review 1 of the empty-responses spec

This review covers the first draft of spec.md, at commit e85e857. It was checked against inspect_ai c2fc3d9 and inspect_evals 6a96ec5. The requester had already decided: a $200 ceiling, no Bedrock, a 16,000-token wide budget, and an open eval list. The spec’s “Changes after review 1” section records what was adopted. Before adopting them, the author spot-checked four claims against the source: Gemini BLOCK_NONE, the FORTRESS judges, CoCoNot max_tokens, and AIR-Bench’s solver-side judge.

Blocking

B1. D1 fires trivially and contradicts H1. D1 triggered when at least 25% of empty replies are truncations, for any reasoning model on any eval. XSTest sets max_tokens=256 (xstest.py:108). gpt-5-mini goes through the Responses API (openai.py:205-215), where max_output_tokens includes reasoning. So nearly every empty reply there will be truncated_empty, which is a fact about the code, not a finding. H1 also used a different threshold, 50%.

Fix: make D1 measure harm, not just existence.

    1. The share of all samples that are truncated_empty at the eval’s own settings. At 5% or more in any eval × model cell, recommend raising max_tokens. At 1% or more, recommend the helper change.
    1. Among truncations re-run under wide, the split between answers, text refusals and still-empty replies.

Align H1 with the same numbers.

B2. The Claude Sonnet 5 “thinking on” arm is under-specified. Claude 5 rejects budget_tokens; thinking is adaptive and controlled by effort (anthropic.py:1413-1423, 1744-1746). Inspect enlarges max_tokens for thinking only when the task leaves it unset (anthropic.py:1588-1625), so XSTest’s 256 is sent as is. The spec should state the exact GenerateConfig for each model.

“The eval’s own settings” is also not a single setting. HLE’s default run config sets no max_tokens, so OpenAI runs at the model maximum while Anthropic runs at 32,000 (anthropic.py:1584). The report should label the setting each route used.

B3. Gemini filtered will be near zero on the direct route. Inspect’s Google provider sets every safety category to BLOCK_NONE by default (google.py:203-222, 270). The direct-versus-OpenRouter pair therefore compares filters off with whatever OpenRouter applies. Say so in the spec, and run the direct route as the evals do.

B4. Part B breaks on AIR-Bench and BFCL.

  • AIR-Bench calls the judge inside its annotate solver. The scorer then reads state.output.completion, which by that point holds the judge’s output (air_bench.py:46, 148-172). The classifier must read the model’s own reply, and the policies must be rebuilt from the stored judge text.
  • In BFCL simple, every correct reply is tool_only with an empty completion (scorer.py:198-253). Treat tool_only as an answer whenever tools were offered.

B5. D3’s safeguard does nothing, and there are too many cells. A paired policy difference changes only on re-routed samples, and they all move one way. So “interval excludes zero” holds as soon as one sample moves. Pre-register the primary cells (XSTest × reasoning models, FORTRESS benign × all, AIR-Bench × all), keep the effect-size threshold, and report every cell.

B6. FORTRESS adversarial judges blow the budget. Its default judges are three models, including openai/o3 and gemini-2.5-pro (fortress.py:21-25). Cut it to 100 + 100 samples in Part A, and to 20 samples × 2 repeats in Part C.

Should fix

  • S1. Part C tests paths that do not exist. XSTest, FORTRESS and HLE never send empty text to their judge. Test empty and whitespace only on unguarded judges. Drop “reasoning shown as text”. State the judge’s temperature.
  • S2. Define what counts as a passing verdict for each eval in D4.
  • S3. The follow-up re-runs are over-built. Keep wide for truncated_empty only, add a 50-reply same-configuration control, and drop the other-route re-run.
  • S4. Two of the D2 pairs are weak. The gpt-oss pair uses two hosts behind one adapter. Mistral has no refusal signal at all. Replace them with Claude Sonnet 5 via OpenRouter. gpt-4.1 (chat completions) against gpt-5-mini (Responses API) already compares two adapters.
  • S5. error_text can misclassify genuine refusals. Anthropic’s canned apology is plain English that a model can produce itself. Require ModelOutput.error to be set (anthropic.py:2089-2127). Confirm that replay never injects NO_CONTENT into stored outputs.
  • S6. Exposure search. DEFAULT_MAX_TOKENS = 2048 applies only to some providers. Also search for fail_on_refusal and react(retry_refusals).
  • S7. Fidelity notes. State that none of the chosen evals use react(), that sample subsets are used, and that settings are recorded on every call.

Nice to have

  • P1 is nearly P0 under value_to_float. Keep it, and say where it differs.
  • Drop SimpleQA Verified; StrongREJECT and CoCoNot cover its role.
  • Say who hand-checks the 50 replies per class, and before which gate.