Validation
An agent that did not write the probe reviewed it on 2026-10-28 (review comment). It reran the tests and the probes at a10cb02 on inspect-ai 0.3.283, including E1 in Docker, and checked every claim in the report against the raw JSON and the eval logs.
| Check | Outcome |
|---|---|
| E2 and E3 tables match their JSON | pass, number for number |
| E1 rerun matches the committed run | pass, apart from container ids and attachment hashes |
| E1’s JSON supports its claims | failed at a10cb02: run A’s re-run and run C’s resume both showed 2 model events, error messages were cut at the first (, and checkpoint failures were attachment refs. Fixed: the JSON now records new and replayed model calls, the resume event, resolved checkpoint errors and the checkpoint files’ triggers. The E1 test asserts on these for runs A, B and C |
| Run B’s explanation | wrong at a10cb02 (“the mock model couldn’t be rebuilt”). Fixed: the model was rebuilt, but its scripted outputs came back as dicts |
| Claims beyond the probe are marked | partly at a10cb02. Fixed: the report now says which resume conditions were tested, cites the docs line behind the bug claim, marks the unscored-threshold point as from the API, and notes that E2’s gap is built into the design |
The UNCHANGED history entry |
explained: ScoreEdit(reason=None) dumped with exclude_none drops reason, which reloads as its default. The reviewer reproduced this, and so did the author |
Fit with docs/framework.md |
gaps at a10cb02 (contract table, stages, RULINGS.md, report format, gate 3 without decision.md). Fixed in the Probes section |
| Repository conventions | pass |
Added after the review, and not re-checked by it: E1 run D (score() on a sandboxed log) and the report section “What a scorer sees”. Run D is asserted in the E1 test. The section’s other rows come from task_state_from_sample and the docs at inspect-ai 0.3.283.