Empty responses: exploration of logs we already hold

October 23, 2026

Exploration

Before the smoke run (spec section 3, “Exploration first”), every log we already hold was scanned for non-answers with this experiment’s classifier. These are exploratory counts. They are not part of any decision rule. Nothing below quotes a question, a reference or a reply.

What was scanned

grader-parsing’s answer-generation logs, fetched from the bucket with uv run python -m eval_experiments.store fetch --manifest experiments/grader-parsing/MANIFEST.json --prefix grader-parsing/logs/sources/ --dest data/exploration. That gives 50 logs, all SimpleQA Verified, with 49,600 replies from 43 models:

  • generated/: 2 logs, 300 samples each. Llama 3.1 8B and GPT-3.5 Turbo via OpenRouter, from Inspect with full model events.
  • hawk/: 48 logs, 1,000 samples each. Frontier and open models at the settings in their file names (_high, _32K, _max and so on), imported from Hawk.

No inspect_evals evaluation-report logs or other Hawk logs were available to this session.

The Hawk imports carry no model events. extract.load_replies reads the reply from the model’s first ModelEvent (RULINGS.md, “The reply is the model’s first generation”). It finds none in an imported log, so it classes all 49,000 Hawk replies as error. For this note only, a script classified those replies from sample.output. That is safe here because SimpleQA Verified does not overwrite the output. The frozen extraction code is unchanged, because every log this experiment writes has events. The 54 error rows below are real sample errors.

Class rates

Class Replies Share of 49,600
answer 49,499 99.80%
error 54 0.11%
silent 34 0.07%
filtered 7 0.01%
truncated_empty 3 0.006%
truncated_text 3 0.006%
any other class 0 0

Logs with at least one non-answer (n = replies in the log):

Log n error filtered silent truncated_empty truncated_text
gemini-3-pro-preview 1,000 11
glm-4-7 (two runs) 2,000 5 21
gemma-4-31b-it 1,000 17
glm-5-1 1,000 5
claude-opus-4-6_32K 1,000 4
claude-opus-4-6_max 1,000 1 3
qwen3-5-plus 1,000 4 1
qwen3-6-max-preview 1,000 4
qwen3-5-flash, qwen3-6-flash 1,000 3 each
qwen3-6-plus 1,000 2 1 1
Qwen3-235B-A22B-Thinking-2507 1,000 2
qwen3-max-2025-09-23 1,000 1
gpt-5-4_xhigh 1,000 3
gpt-5-nano_high, gpt-5-1_high 1,000 2 each
gpt-5_high, gpt-5-mini_high, gpt-5-2_xhigh 1,000 1 each
kimi-k2-thinking-turbo 1,000 1
generated, Llama 3.1 8B 300 1

The other 29 logs have no non-answers.

Examples per class

Each example gives only the fields the classifier reads.

  • filtered: OpenAI’s GPT-5 family, on a benign factual eval. All seven are GPT-5 models (gpt-5, gpt-5-mini, gpt-5-nano, gpt-5.1, gpt-5.2) with stop_reason=content_filter and no text. Reasoning tokens run from 192 to 1,152. Three show a negative output_tokens (−11 to −54), so the provider’s usage reported fewer output tokens than reasoning tokens. SimpleQA asks factual questions, so these are filter refusals on prompts that are not harmful. This is the form D2 counts as a provider-signalled refusal.
  • silent: Gemini 3 Pro preview, reasoning then nothing. All 11 have stop_reason=stop, no text and no reasoning content, but 567 to 2,380 reasoning tokens. The model reasoned and then wrote nothing, and since the reasoning text is absent the classifier calls this silent, not reasoning_only. Two Qwen replies look the same (stop, 24,512 and 79,035 reasoning tokens, no text).
  • silent: GLM-4.7, nothing at all. 21 replies with stop_reason=unknown, no text and no usage recorded.
  • truncated_empty: Claude Opus 4.6 at maximum effort. Three replies used all 32,000 output tokens (reasoning tokens 6,953 to 9,978) and stopped on max_tokens with no text. That is 0.3% of that run, on a short-answer eval.
  • truncated_text. Qwen3.6 Plus and Kimi K2 Thinking each wrote over 200,000 characters before hitting max_tokens, and Llama 3.1 8B hit 16,384 tokens with 45 characters of text.
  • error: provider content checks. 19 of the 54 are Alibaba-hosted Qwen models answering HTTP 400 data_inspection_failed, “Input/Output data may contain inappropriate content”. Here a provider refuses through an API error, so the sample fails, on a benign eval. GLM-4.7 adds 5 LengthFinishReasonErrors, where the SDK could not parse a reply cut off at the length limit; that is a truncation that arrives as an error. The other 30 are timeouts, connection errors, rate limits and one IndexError.

What this suggests for the smoke run and the main run

  • On a capability eval at the models’ usual settings, non-answers are rare: 0.2% overall, at most about 2% in a single run (Gemma’s errors). Part A’s own evals need their provocations (256-token limits, harmful prompts) to give D1-D3 enough cases, as the spec assumed (section 7).
  • All four forms the classifier separates turn up in the wild: provider filter (filtered), provider refusal as an API error (error), reasoning with no text (silent), and truncation with no text (truncated_empty). OpenAI and Alibaba both refuse benign prompts by these routes, which supports looking at more than completion.
  • reasoning_only depends on the provider returning reasoning text. When reasoning tokens are billed but the text is not returned (as in these Gemini 3 Pro logs), the reply is classed silent. This touches only the exploratory reasoning_only versus silent comparison (spec section 6). The smoke run checks what the Gemini 3.8 Flash and GPT-5.4 mini routes return.
  • A truncation can arrive as a sample error (GLM-4.7’s LengthFinishReasonError) and not as truncated_empty. P3 already routes both error and truncated_empty as unscored, so the result is the same, but D1(a) counts only truncated_empty. The smoke and main runs report the error_kind of every error, so any such case is visible.