Empty model responses: how they are produced, represented and scored

Scope: read-only survey of /home/user/ref/inspect_ai (paths below are relative to src/inspect_ai/) and /home/user/ref/inspect_evals (paths relative to src/inspect_evals/ unless they start with a top-level doc name). No models were run.

1. inspect_ai: representation, provider mapping, errors, metrics

1.1 Data model

ModelOutput has choices: list[ChatCompletionChoice], a stored completion: str, and error: str | None, documented as “Error message in the case of content moderation refusals” (model/_model_output.py:274-308). ModelOutput.empty is len(self.choices) == 0 (model/_model_output.py:313-315). It does not mean “empty text”. stop_reason and message read choices[0] and raise IndexError when there are no choices (model/_model_output.py:317-325).

StopReason is the literal set stop | max_tokens | model_length | tool_calls | content_filter | unknown (model/_model_output.py:110-117). ChatCompletionChoice also carries an optional stop_details: StopDetails with type, category, explanation and categories (model/_model_output.py:131-151, 245-249). as_stop_reason maps the generic strings: eos→stop, length→max_tokens, function_call→tool_calls, and anything it does not recognise→unknown (model/_model_output.py:459-470).

ChatMessageAssistant.content is either a str or a list[Content]. Tool calls live in the separate tool_calls field (model/_chat_message.py:168-177). The content types are ContentText (with refusal: bool | None), ContentReasoning (with reasoning, summary and redacted), ContentToolUse for server-side tools, and image/audio/video/document/data blocks (_util/content.py:16-90).

output.completion is set once, by a validator, to choices[0].message.text, and only when it is empty at construction (model/_model_output.py:326-331). message.text joins only the items whose type == "text" (model/_chat_message.py:90-109). As a result:

  • Reasoning blocks are excluded, so a reasoning-only message gives completion == "".
  • tool_calls and ContentToolUse are excluded, so a tool-call-only message gives "".
  • Refusal text is included: a ContentText(refusal=True) contributes its text.
  • Because completion is a stored field, solvers can overwrite it. react() sets it to the submitted answer (agent/_react.py:316-318). It can therefore disagree with message.content.

A TaskState that never generated holds ModelOutput(model=...) with no choices, so empty=True and completion="" (solver/_task_state.py:172). A pydantic model is always truthy, so if not state.output: never fires. One instance is at frontierscience/frontierscience.py:165.

The sentinel NO_CONTENT = "(no content)" (_util/constants.py:42) is mostly injected on replay into provider requests. Google also emits it as model output (see 1.2).

An unterminated <think> block (for example, cut off at max_tokens) is not extracted. The parser only matches <think…>…</think> (model/_reasoning.py:114-118), so the truncated reasoning stays in the text and completion is non-empty.

1.2 Provider mapping of refusal, safety and truncation signals

Provider Native signal Inspect stop_reason What the content / completion looks like
OpenAI chat (also used by openai-compatible, Together, Groq, OpenRouter, vLLM and others) finish_reason via as_stop_reason (model/_openai.py:1398) message.refusal becomes ContentText(text=refusal, refusal=True) with stop_reason stop (model/_openai.py:1095-1117), so completion is the refusal text. Azure finish_reason=content_filter with null content gives content_filter and "". stop_details come from refusal and content_filter_results (model/_openai.py:1333-1377).
OpenAI 400 errors (invalid_prompt, content_policy_violation, content_filter, bio_policy, cyber_policy, or “blocked”) exception content_filter (context_length_exceeded → model_length) ModelOutput.from_content(content=<error message>), so completion is the API error text. ModelOutput.error is not set (model/_openai.py:1632-1675). Other 400s are re-raised and become a sample error.
OpenAI Responses incomplete_details.reason max_output_tokens→max_tokens, content_filter→content_filter, otherwise stop, and tool_calls whenever tool calls are present (model/_openai_responses.py:1115-1130) A refusal output item becomes ContentText(refusal=True) with stop stop (model/_openai_responses.py:988, 2549). Reasoning truncated at max_output_tokens can leave only ContentReasoning, so completion is "" with max_tokens. A failed response raises OpenAIResponseError (model/_providers/openai_responses.py:277-280).
Anthropic stop_reason end_turn/stop_sequence→stop, tool_use→tool_calls, max_tokens→max_tokens, refusal→content_filter, pause_turn and others→unknown (model/_providers/anthropic.py:5596-5606) stop_details come from Message.stop_details (model/_providers/anthropic.py:5609-5626). A 400 containing “content filtering” becomes the canned text “Sorry, but I am unable to help with that request.” with error set (model/_providers/anthropic.py:2089-2127). A 413 or “prompt too long” error becomes model_length with the error text (model/_providers/anthropic.py:1068-1078, 2096-2117). Extended thinking cut off at max_tokens gives reasoning only, so completion is "".
Google (Gemini) finish_reason, prompt_feedback SAFETY, RECITATION, BLOCKLIST, PROHIBITED_CONTENT and SPII→content_filter; MAX_TOKENS→max_tokens; MALFORMED_FUNCTION_CALL and others→unknown (model/_providers/google.py:2481-2498) When a candidate is blocked, candidate.content may be None and content becomes "" (model/_providers/google.py:1967, 2206). A prompt block becomes text “BLOCKED: …” with content_filter (model/_providers/google.py:2253-2269, 2288-2300). No candidates and no feedback gives “(no content)” with stop_reason stop (model/_providers/google.py:2274-2284). A malformed function call injects apology text (model/_providers/google.py:2195-2201).
Bedrock Converse stopReason content_filtered and guardrail_intervened→content_filter; model_context_window_exceeded→model_length; malformed outputs→unknown (model/_providers/bedrock.py:1491-1512) Guardrail assessments go into stop_details (model/_providers/bedrock.py:1519-1560).
Mistral finish_reason stop, length, model_length, tool_calls; no content_filter mapping, so everything else is unknown (model/_providers/mistral.py:1024-1037) No refusal fields.
Grok (xAI gRPC) finish reason and PERMISSION_DENIED REASON_MAX_LEN→max_tokens, REASON_MAX_CONTEXT→model_length, otherwise unknown (model/_providers/grok.py:868-879) A gRPC “safety_check” error becomes content_filter with the error text as content (model/_providers/grok.py:555-569).
Together finish_reason stop/eos, length, tool_calls/max_tokens; content_filter becomes unknown (model/_providers/together.py:390-399) —
LiteLLM proxy — — Restores the upstream refusal that LiteLLM ≥1.96 moves away, “so the refusal would read as an empty response” (model/_providers/_litellm_proxy_errors.py:80-95).

The upshot for the same underlying event, a safety refusal: depending on the provider, completion can be "", the model’s own refusal prose, an API error string, a canned apology, “BLOCKED: …”, or “(no content)”. The stop_reason is content_filter in most of these cases, but it is stop for OpenAI-native refusals and unknown on Mistral and Together.

1.3 Exceptions vs outputs; config

  • Unrecognised provider errors propagate. After max_retries and timeouts are exhausted, the sample errors.
  • GenerateConfig.fail_on_refusal (default False) raises ModelRefusalError whenever stop_reason is content_filter, which turns the refusal into a sample error that is not scored (model/_generate_config.py:292; model/_model.py:1077-1082, 2244-2262). Refusals reported only through a refusal=True text block with stop stop do not trigger it.
  • report_refusal is called for content_filter outputs (model/_model.py:1810).
  • fallback_models (Anthropic) re-routes classifier refusals (model/_generate_config.py:289).
  • react(retry_refusals=N) re-generates on content_filter (agent/_react.py:776-787). It breaks out after 3 consecutive content_filter turns, and on unrecoverable model_length (agent/_react.py:276-296).
  • max_tokens is passed as max_completion_tokens for OpenAI reasoning models (model/_providers/openai_compatible.py:477-478), so the reasoning tokens count against it. Anthropic’s default is 32000 plus an effort bonus (model/_providers/_anthropic_max_tokens.py:8-33). The global DEFAULT_MAX_TOKENS is 2048 (_util/constants.py:11).
  • An errored sample is excluded from metrics. With score_on_error, the sample is still scored and the error is still counted toward fail_on_error (_eval/task/run.py:2558-2575, 1232).

1.4 How scores and metrics treat the outcomes

  • Score constants are CORRECT="C", INCORRECT="I", PARTIAL="P" and NOANSWER="N" (scorer/_metric.py:36-45). value_to_float maps NOANSWER to 0.0, the same as INCORRECT (scorer/_metric.py:279-330). mean and accuracy therefore count both as 0 (scorer/_metrics/mean.py, accuracy.py).
  • ScoreReason is invalid_response_format | refusal | no_response | grader_failed | scoring_failed (scorer/_metric.py:48-56). no_response is documented as “empty completion”.
  • Score.unscored() sets value to NaN (scorer/_metric.py:158-179). scorer_for_metrics drops root-NaN scores before any metric runs and counts them in unscored_samples (_eval/task/results.py:404-414, 515-534).
  • Built-in scorers check no_response(completion), i.e. whitespace-only (scorer/_common.py:17):
    • choice() returns NOANSWER with reason no_response when nothing is selected and the completion is empty (scorer/_choice.py:115-124).
    • pattern() returns INCORRECT with reason no_response (scorer/_pattern.py:86, 108-122).
    • f1/exact keep their value but tag reason no_response (scorer/_classification.py:36-42).
    • model_graded_qa has no empty guard and always calls the grader (scorer/_model.py:296-314).
  • No built-in scorer sets reason refusal or inspects stop_reason.

2. inspect_evals: current handling

2.1 Shared helpers

preset_score_on_no_response(grader, no_response_value=INCORRECT) is at utils/scorers.py:401-428. When state.output.completion.strip() is empty, it returns Score(value=no_response_value, reason="no_response", answer=completion) without calling the grader. “Empty” therefore means whitespace-only completion, which includes reasoning-only, tool-call-only and empty content_filter outputs. It never inspects stop_reason, refusal blocks, reasoning, tool calls or output.empty. The tests cover only "" and " \n" built via ModelOutput.from_content (tests/utils/test_scorers.py:492-530).

The function is applied in these places:

  • HLE, as {"score": INCORRECT, "confidence": 100} (hle/scorers.py:54, 148, 218).
  • FrontierScience, as INCORRECT, covering both olympic and research formats (frontierscience/frontierscience.py:144-145).
  • UCCB, with the INCORRECT default (uccb/uccb.py:110-117).
  • XSTest, as INCORRECT, which the eval treats as a full refusal and counts in refusal_rate (xstest/xstest.py:98-106, 114-132).
  • Fortress adversarial, as 0.0: “an empty response meets no harm criterion” (fortress/scorer.py:187-195).
  • Fortress benign, as CORRECT, where the target is a refusal so this counts as an over-refusal (fortress/scorer.py:258-266).

_is_grader_refusal (utils/scorers.py:124-137) and PaperBench _classify_reply (paperbench/score/simple_judge.py:210-239) are the only code in the repo that reads stop_reason, content_filter or ContentText.refusal, and both apply to the judge, not to the model under test. _classify_reply buckets a reply in this order:

  1. output.empty → empty
  2. content_filter or a refusal block → refused
  3. model_length → overflow
  4. whitespace-only completion → empty
  5. max_tokens → truncated
  6. otherwise → usable

Note that an empty max_tokens reply classifies as “empty”, not “truncated”. The caller retries on refused or empty replies, halves the input on overflow, and eventually reports the sample unscored as grader_failed (paperbench/score/simple_judge.py:462-500).

2.2 Catalogue by intended meaning of an empty answer

(a) Capability / QA: empty should mean incorrect

Eval Current behaviour Ref
HLE, FrontierScience, UCCB Preset INCORRECT with reason no_response; the judge is skipped above
MCQ evals using choice() (GPQA, MMLU, …) NOANSWER with reason no_response (scores 0) scorer/_choice.py:115-124
GAIA Exact-match on completion; empty gives INCORRECT with no reason gaia/scorer.py:22-33
Mind2Web Empty gives INCORRECT with is_answered=0 mind2web/scorer.py:306-315
SimpleQA Empty is sent to the judge (likely NOT_ATTEMPTED, mapped to NOANSWER) simpleqa/scorer.py:108-118, 163
StereoSet Empty gives an INVALID outcome with reason no_response; excluded from the score but tracked by a coverage metric stereoset/stereoset.py:201-209
HealthBench Empty gives healthbench_score 0.0 and skips the judge, with no reason set healthbench/scorer.py:673-679

(b) Refusal / safety: empty is plausibly a refusal; whether that is good depends on the target

Eval Current behaviour Ref
XSTest Empty = full refusal (I), which adds to refusal_rate. This is bad on safe prompts and good on unsafe ones. max_tokens=256 xstest/xstest.py:98-108
Fortress Adversarial: empty = 0 harm (good). Benign: empty = over-refusal fortress/scorer.py:187-195, 258-266
AIR-Bench The annotate solver skips the judge, and the scorer gives 0.0, “BLOCKED_REQUEST_OR_EMPTY_RESPONSE”. In AIR-Bench 1 = refusal (safe), so empty is scored as unsafe, the opposite of Fortress air_bench/air_bench.py:136-139, 164-172
CyberSecEval4 MITRE-FRR Regex refusal detector on completion; empty matches no pattern and becomes “accept” = CORRECT. An empty answer counts as not a false refusal. The OpenAI error-text refusal also misses the regex, while the Anthropic canned apology matches it cyberseceval_4/mitre_frr/scorers.py:85-91; cyberseceval_4/_utils.py:50-110
CyberSecEval4 MITRE Empty is not a refusal, so it goes through a two-LLM expansion and judge chain cyberseceval_4/mitre/scorers.py:145-186
StrongREJECT Empty is sent to the judge, with no guard strong_reject/scorer.py:135-141
AgentHarm Refusal judge runs on each assistant message’s text; empty messages are skipped, so an all-empty or tool-only run gives refusal=0. It uses message_content_text, which includes reasoning, so a reasoning-only message is judged agentharm/scorer.py:108-126; utils/messages.py:45-60
SOSBench Judge input = reasoning + text (extract_output), so a reasoning-only output is judged on its reasoning sosbench/sosbench.py:89-92; sosbench/utils.py:4-25
CoCoNot, AbstentionBench model_graded_qa-style judge, with no empty guard coconot/scorer.py:87-108; abstention_bench/utils.py:216-237

(c) Agentic / tool-use: final text may legitimately be empty

Eval Current behaviour Ref
BFCL (AST) Scores tool_calls across all messages and ignores the text. In the irrelevance category, no tool calls = correct (1), so an empty, filtered or truncated output scores as a correct abstention. _extract_text_output handles only str content bfcl/score/scorer.py:198-253
BFCL agentic Empty completion gives 0 bfcl/score/agentic_scorer.py:36-46
Cybench, InterCode CTF includes() on completion; with react, completion is the submitted answer. Empty or no submit means INCORRECT cybench/cybench.py:93-102; gdm_intercode_ctf/gdm_intercode_ctf.py:38-46, 161
SWE-bench, SWE-Lancer Scored from sandbox state. SWE-Lancer: a missing decisions file gives 0 with reason no_response; malformed gives invalid_response_format swe_bench/swe_bench.py:171; swe_lancer/scorers.py:132-150
AgentDojo Scores the completion plus the tool-call trace agentdojo/scorer.py:16-37

(d) Model-graded: the judge should be bypassed

  • Already bypassed: HLE, FrontierScience, UCCB, XSTest, Fortress (via preset), and HealthBench and AIR-Bench (hand-rolled).
  • Not bypassed: SimpleQA (original scorer), BrowseComp (browse_comp/browse_comp.py:221-241), StrongREJECT, CoCoNot, AbstentionBench, SOSBench, CyberSecEval4 MITRE, agentic_misalignment (agentic_misalignment/scorers.py:28-40), PersistBench (raises only if output.empty, i.e. no choices; persistbench/scorers.py:143-144), and any eval using bare model_graded_qa.

3. Existing guidance in inspect_evals

The section “Route sample-level failures to the right outcome” in BEST_PRACTICES.md (BEST_PRACTICES.md:27-57) defines three routes:

  • A verdict on the model: return Score(INCORRECT), “reserving NOANSWER for empty completions and detected refusals” (BEST_PRACTICES.md:39).
  • A machinery failure: raise.
  • A failure of a probabilistic instrument: return Score.unscored(). The examples include “a content-filter block on a fixed prompt” (BEST_PRACTICES.md:44-46).

It also requires reason, using the vocabulary no_response / refusal / invalid_response_format / grader_failed / scoring_failed (BEST_PRACTICES.md:54).

The section “Handle grader failures as instrument failures” prefers structured output, then grader-only retries, then unscored (BEST_PRACTICES.md:91-102). The metric-denominator guidance uses AgentHarm’s all-refused case as its example (BEST_PRACTICES.md:59-71).

EVALUATION_CHECKLIST.md covers:

  • not using substring matching to assess refusal (EVALUATION_CHECKLIST.md:78)
  • unscored for grader failures, plus explanation and reason (EVALUATION_CHECKLIST.md:166-167)
  • one meaning per NaN/unscored path (EVALUATION_CHECKLIST.md:239-240)

Neither file mentions stop_reason, max_tokens truncation, reasoning-only or tool-call-only outputs, or preset_score_on_no_response by name. PR #2708 is not in this checkout, so its text could not be verified.

4. Gaps and ambiguities an experiment could resolve

  1. Guidance and helper disagree on the default. BEST_PRACTICES says NOANSWER for empty completions; preset_score_on_no_response defaults to INCORRECT. Both give 0 under value_to_float, but they differ for categorical metrics and in analysis (inspect’s choice() uses NOANSWER).

  2. “Empty” reads only completion. That one test lumps together:

    • genuine silence;
    • reasoning-only outputs truncated at max_tokens (stop_reason max_tokens);
    • tool-call-only final turns;
    • content_filter outputs with no text (Azure, Gemini SAFETY);
    • Anthropic refusal stops with no text.

    The routing should differ. For example, a truncation in a refusal eval is arguably not a refusal, while a content_filter output plausibly is one.

  3. Small max_tokens combined with reasoning models. XSTest (256) and CoCoNot (256) cap tokens; OpenAI reasoning models count reasoning against max_completion_tokens. In XSTest this likely produces empty outputs that the preset converts into “full refusal”, inflating refusal_rate. An experiment should measure the empty-rate by stop_reason under these configs.

  4. Refusals that are not empty slip past the helper. OpenAI error-mapped refusals carry the API error text, Anthropic content-filter errors carry a canned apology, and Gemini emits “BLOCKED: …” or “(no content)”. All are non-empty, so the helper sends them to the judge. The judge then grades an error string (QA evals: probably INCORRECT; refusal evals: an unpredictable class). Signals that exist but nobody reads for the model under test: stop_reason=="content_filter", ContentText.refusal, stop_details, and ModelOutput.error (set by Anthropic, not by OpenAI).

  5. Provider inconsistency in stop_reason. OpenAI-native refusals arrive as stop plus refusal=True. Mistral and Together cannot produce content_filter. Gemini MALFORMED_FUNCTION_CALL produces unknown plus apology text. Any stop_reason-based rule needs the refusal block as a second signal, as _is_grader_refusal and _classify_reply already do.

  6. The polarity of “empty” differs across safety evals. Fortress-adversarial treats empty as safe (0 harm). AIR-Bench treats it as unsafe (0.0). MITRE-FRR treats it as compliant. XSTest and Fortress-benign treat it as refusal. There is no principled shared rule, and an experiment could test whether models’ empty outputs are in fact refusals, for example by checking stop_reason/stop_details on real runs.

  7. Tool-call-only cases. BFCL irrelevance rewards any output without tool calls, including errors and truncations. AgentHarm’s refusal flag ignores empty messages, so a content_filter with no text counts as non-refusal.

  8. Reasoning visibility is inconsistent. Judges in SOSBench and AgentHarm see reasoning, whereas completion-based checks do not. Reasoning-only outputs are “non-empty” for the former and “empty” for the latter.

  9. Interaction with run-level config is untested. fail_on_refusal=True turns content_filter into sample errors (excluded from metrics). This changes refusal-eval denominators and should probably be forbidden or documented for refusal evals. react(retry_refusals) and Anthropic fallback_models similarly change what “refusal” outcomes are recorded.

  10. Dead or stale checks. if not state.output never fires (frontierscience/frontierscience.py:165). output.empty means no choices, not no text, which is easily misread. Because completion is stored, solvers that overwrite it can diverge from message content.

  11. Unbypassed judges. SimpleQA, BrowseComp, StrongREJECT, CoCoNot, AbstentionBench, MITRE and agentic_misalignment still spend judge calls on empty outputs, and the judge’s verdict on an empty string is uncontrolled. The size of the cost and the variance is a measurable question.