Scoring recovery: results

Run on 2026-10-28 with inspect-ai 0.3.283 and the mock model. The raw outcomes are in e1.json, e2.json and e3.json. An earlier, uncommitted run on 0.3.285 gave the same outcomes.

E3: score() on samples whose scorer raised

score() is the Python function behind inspect score. Four samples, and the judge raises on samples 2 and 3 (fail_on_error=False).

Step	Accuracy	Scored samples	Samples 2 and 3
eval	1.0	2	errored, no score
score(..., action="overwrite") with a working judge	1.0	4	scored C, still errored
the same, with the judge failing on sample 2	n/a	n/a	score() raised RuntimeError, nothing returned

E2: re-judging only grader_failed samples

Six samples. The judge logs the answer as metadata["evidence"], then fails on samples 2, 3 and 5 with Score.unscored(reason="grader_failed"). Samples 3 and 5 hold the only wrong answers, so the outage hides both of them by construction. True accuracy is 4 of 6, 0.667.

Step	Accuracy	Stderr	Scored	Unscored
eval, during the outage	1.000	0.000	3	3
after re-judging samples 2, 3 and 5 from the logged evidence	0.667	0.211	6	0

Sample 2’s Score.history after a write and a re-read:

Entry	Value	Reason	Author
original	nan	grader_failed	none
edit	C	UNCHANGED	scoring-recovery

The edit set reason=None, and the current score’s reason reads None, but the history entry shows UNCHANGED. ScoreEdit(reason=None).model_dump(exclude_none=True) drops reason, and it reloads with its default, "UNCHANGED". The evidence in metadata survived the edit.

E1: checkpoint resume for scoring, Docker sandbox

A react() agent writes ~/answer.txt (42) and starts a web server on port 8000. The scorer reads the file, probes the server and raises on its first call. Checkpointing uses TurnInterval(every=1) and retention="retain". In every first attempt, the checkpoint files have triggers turn and agent_complete.

Run	How the sample was retried	Resume recorded	New model calls	Replayed model calls	Scorer’s second call saw	Outcome
A	in-run retry_on_error=1	none	2	0	file 42, server up, new container	C
B	eval_retry(log)	none	1	0	no second call	ValueError
C	eval_retry(log, checkpoint=same_config)	resume_for_scoring	0	2	file 42, server down, new container	C

• A. The retry re-ran the agent, which wrote the file and started the server again. Both of the retry’s checkpoint writes failed with EgressVerificationError: sandbox egress ckpt-00003 -> ... is only accepted while the destination repo is uninitialized. The retry wrote to the same restic repository as the first attempt.
• B. The retry re-ran the agent. Its first model call failed with ValueError("output must be an instance of ModelOutput; got <class 'dict'> ..."): the mock model was rebuilt from the log, but its scripted outputs came back as dicts.
• C. The sample resumed with an info event {"event": "resume", "attempt": "resume_for_scoring"}. The two model calls in its log are the first attempt’s, replayed inside a prior_run span.
• D. score(c1_log, sandbox_judge()) on run C’s first, errored log, with the scorer that reads the sandbox, raised ProcessLookupError: No sandbox environment has been provided for the current sample or task. Re-scoring from a log has no sandbox.