When a grader fails, can the model’s response be scored later?

October 28, 2026 · probe · draft

Question
When a grader fails on a sample, which Inspect mechanisms keep the model’s response so the sample can be scored later, and what does each need?
Decision
Unscored is recoverable only when everything the grader needs is in the log; a sandbox scorer that does not log its evidence needs raise plus checkpoint resume.
Feeds
inspect_evals#2599, inspect_evals#2708
Run
3 probes on Inspect’s mock model and a local Docker sandbox, inspect-ai 0.3.283
Spend
$0
Detail
full results validation question

All results are exploratory, from a probe using Inspect’s mock model and a local Docker sandbox, on inspect-ai 0.3.283.

Headline

A failed grade can be redone without re-running the model, but how depends on what the scorer reads: only what the log holds, or the sandbox too.

  • A scorer that reads only what the log holds loses nothing either way. That means the TaskState fields a log keeps: messages, output, store and metadata (see below).
    • If it raises, score() (inspect score) can score the errored samples later from the log. They keep their error flag, every sample is re-graded, and one remaining grader failure aborts the whole re-score (E3).
    • If it returns Score.unscored(reason="grader_failed"), a short script using edit_score() and recompute_metrics() can re-judge just those samples (E2).
  • A scorer that calls sandbox() can’t be re-run from the log. score() on such a log raised ProcessLookupError: No sandbox environment has been provided (E1, run D). It has two ways out:
    • write the evidence it collects from the sandbox to the store (or the score’s metadata) before calling the judge, so a re-judge needs only the log;
    • resume through checkpointing.
  • Checkpoint resume. eval_retry resumed “for scoring” with no new model calls when it was given the same checkpoint= config as the original run. Without that config, it re-ran the agent. Files under the sandbox paths came back, but a server the agent left running was down when the scorer ran (E1, runs B and C).
    • Only the config condition was tested.
    • The docs also require an agent that supports checkpointing, and kept checkpoints. Every run here used react() and retention="retain".
    • The eval_retry docstring says the config must be passed again. The guide’s recovery example (inspect eval-retry logs/<log>.eval) leaves it out.
  • In-run retry_on_error re-ran the agent even with checkpointing on, and the retry’s checkpoint writes failed with EgressVerificationError (E1, run A). The checkpointing docs say “When Inspect retries a failed sample, checkpointed data is automatically restored”, so this looks like an Inspect bug.
  • Unscoring can hide an outage. With the judge failing on the 3 samples that held both wrong answers, accuracy read 1.000 over 3 samples against a true 0.667. Only the unscored count showed it. The gap is built into the design, but the mechanism is general.

What this means for #2599 Q2. “Unscored” is only recoverable when everything the grader needs is in the log. That covers scorers that read only messages, output, store and metadata, and sandbox scorers that write their evidence to the store before calling the judge. For a sandbox scorer that doesn’t, raising plus checkpointing is the only way tested here to keep the agent’s work. Either way, a run needs a coverage check. From the API rather than this probe: Inspect has no threshold on unscored samples like fail_on_error.

How far this matches real use. Failures were simulated by scorers that raise or unscore on chosen samples, and the model was scripted. Real graders, real providers and sandbox providers other than Docker weren’t tested. Checkpointing fired every turn, which a real run would not do, and it was configured on the eval rather than per task or sample.

What a scorer sees

A scorer gets a TaskState and a Target, and can reach sandbox(), get_model() and model roles, and transcript() through context. Where the TaskState comes from depends on how scoring started. The cells that cite run C or D come from this probe. The rest come from the code and docs at inspect-ai 0.3.283.

When scoring runs Where the TaskState comes from Sandbox
during the eval the solver chain’s final state, every field live live: the agent’s container, processes included
score() / inspect score rebuilt from the logged sample by task_state_from_sample: input, target, choices, messages, output, metadata, store and uuid, with completed=True, earlier scores only in append mode, attachments resolved, and transcript() seeded with the logged events and timelines. Tools are empty and limits unset none: sandbox() raises (run D)
checkpoint resume_for_scoring a fresh sample from the dataset (input, target, metadata) plus the agent’s tracked state (react() tracks messages and output), with the store and events restored a fresh container with sandbox_paths restored, processes lost (run C)
score_on_error whatever partial state the solver reached before the error live

Most built-in and inspect_evals scorers read state.output.completion (match(), includes(), choice(), model_graded_qa(), which also uses input_text). Agentic scorers often read messages or the store. Sandbox scorers call sandbox() to run tests, read files the agent wrote, or check a flag. The test that matters for re-scoring is whether the scorer calls sandbox() or anything else outside the TaskState, not whether the eval is agentic: a scorer that compares output against a file in the sandbox can’t be re-run from the log.

What was done

Three probes, each a short script in src/eval_experiments/scoring_recovery/:

  • E3: an eval where the judge raises on 2 of 4 samples, then score() over the log with a working judge, then with a judge still failing on one sample.

  • E2: a judge that logs the answer as evidence and unscores 3 of 6 samples. A loop then re-judges only the grader_failed samples from the evidence.

  • E1: a react() agent that writes a file and starts a server in a Docker sandbox, with a scorer that fails on its first call. Run three ways:

    • A, in-run retry;
    • B, eval_retry alone;
    • C, eval_retry with the checkpoint config passed again;
    • D, score() on C’s errored log with the same scorer.

    The probe tells fresh model calls from ones a resume replayed, by whether they sit inside a prior_run span.

Results

See results/main/summary.md, with the raw outcomes in the JSON files beside it.

What the results don’t show

  • Whether checkpoints are deleted when retention isn’t set and the eval ends as success with errored samples. The docs say they are, but every run here kept them.
  • How long a real scoring-only resume takes, or what its checkpoints cost to store.
  • Whether a resume works with a custom agent, or with a checkpoint config set per task or sample.

Possible upstream reports

  • In-run retry_on_error with checkpointing re-runs the agent, and the retry’s checkpoint writes fail with EgressVerificationError.
  • The inspect eval-retry recovery example in the checkpointing guide doesn’t mention passing --checkpoint again.
  • ScoreEdit(reason=None) loses reason when saved, so the history shows UNCHANGED.
  • inspect score can’t target selected samples, and aborts on the first scorer failure.

Reproduction

uv sync
uv run python -m eval_experiments.scoring_recovery.probes --out experiments/scoring-recovery/results/main
# without Docker, run only E2 and E3
uv run python -m eval_experiments.scoring_recovery.probes --out experiments/scoring-recovery/results/main --only e2 e3