When is an empty response an error, a wrong answer, unscored, or a success?

October 22, 2026

Question

When the model under test returns what looks like an empty response, how should an inspect_evals scorer score it: as an error, as incorrect, as unscored, or (in an eval that tests refusal) as a success? And how should a scorer tell the different kinds of “empty” apart?

Background

inspect_evals#2415 reported that FORTRESS and HLE sent samples to their LLM judge even when the model under test had given no response, so the judge graded nothing. #2669 (merged 2026-10-13) added preset_score_on_no_response, a scorer wrapper that skips the judge and returns a preset score when state.output.completion.strip() is empty. It is applied to HLE, FORTRESS, UCCB, XSTest (empty counts as a full refusal) and FrontierScience (empty counts as incorrect). In review, it was suggested that refusals be handled the same way, following PaperBench’s _classify_reply. #2708 (open) documents failure routing in BEST_PRACTICES.md.

That fix answers one case. It leaves open:

  • What “empty” means. completion is the text of the reply. A reply that is only a tool call, or only reasoning, or reasoning cut off at the token limit, also has empty completion, but these are different events from a model that returned nothing.
  • What an empty reply means for the eval. For a capability eval it is plausibly a wrong answer. For an eval of whether a model refuses harmful requests, it may be a refusal and so a success, or a failure to engage, depending on what the eval measures. For an agentic eval, a final turn with no text may be normal.
  • How providers signal refusal. Some return a refusal as text, some as a separate refusal field, some as a content_filter stop reason with no text, and some as an API error. The same refusal can therefore reach a scorer as text, as an empty reply, or as a sample error, depending on the provider.
  • What counting it each way does to the headline number. Scoring empty as incorrect lowers accuracy. Leaving it unscored removes it from the denominator. Counting it as an error stops the sample or retries it. Which is right depends on whether the empty reply is the model’s behaviour or a fault in the pipeline.

Decision this informs

What inspect_evals says, in BEST_PRACTICES.md and in shared helpers, about routing empty and refusal outputs:

  1. Whether preset_score_on_no_response (or a successor) should look at more than completion, such as the stop reason, tool calls, reasoning content and the provider’s refusal signals, and classify the reply before choosing a score.
  2. What the default route should be for each class of empty reply, and which classes each eval must decide for itself, given what the eval measures.
  3. Whether truncation (a reasoning model hitting the token limit) should be an error, retried, or scored, and whether evals should set a minimum token budget for reasoning models.

The decision is made by the inspect_evals maintainers. This experiment supplies evidence and a recommended rule.

What is already known

  • Inspect records a stop_reason on every model output, and the provider adapters map provider-specific signals into it. How consistently they do so for refusals and truncation is part of what the experiment will check.
  • PaperBench already classifies replies before scoring (_classify_reply).
  • No measurement exists, as far as we know, of how often each kind of empty reply occurs across providers and evals, or how much the choice of route moves published scores.

Out of scope

  • Whether an eval’s refusal criterion is right in itself (for example, what XSTest should count as a refusal).
  • Judge-side failures, where the model under test answered and the judge returned nothing. grader-parsing covered those.
  • Changing Inspect AI. If the evidence points to a change there, the report says so and a separate proposal follows.