Question

When a grader fails on a sample, which Inspect mechanisms keep the model’s response so the sample can be scored later, and what does each need?

Decision it informs

Q2 of inspect_evals#2599: should a grader that fails in a possibly transient way (an unreachable remote judge, an API timeout) raise, or return Score.unscored(reason="grader_failed")? The decision on 16 October was to raise, following the maintainers’ rule that a failed grader call is machinery. On #2708 Matt reopened it: a long agentic sample whose model has finished shouldn’t be errored and re-run because the grader timed out. The inspect_evals maintainers decide.

What was known

  • Raising errors the sample. retry_on_error and eval_retry re-run the whole sample, model included.
  • Score.unscored() keeps the sample in the log but out of the metrics. eval_retry and eval_set treat it as complete.
  • inspect score re-scores a whole log. It has no option to choose samples.
  • edit_score() and recompute_metrics() can change individual scores in a log.
  • Checkpointing (inspect_ai#4078) writes an agent_complete checkpoint, and a later retry can resume “for scoring” with no model calls. It needs a checkpoint-aware agent.
  • A scorer that reads the sandbox can’t be re-run from the log, since the log doesn’t hold the sandbox.

Out of scope

  • Real graders and providers. The probes simulate failures.
  • Which failures count as transient. That’s for the policy, not this probe.
  • Sandbox providers other than Docker.