Agent-run experiments: design

October 22, 2026

This document describes how an agent takes an open question about evals and turns it into a finished experiment and a report that justifies a decision. It covers the lifecycle, the human gates, the files every experiment must have, the shared code, how results are checked, and the report format. It is written from how grader-parsing and grader-prompt-structure were actually run, and is meant to be reviewed before any of it is built.

Goal

Given a pointer to a question (a GitHub issue, PR or comment, or a prompt), an agent should:

State the question and the decision it informs.
Design an experiment and pre-register it.
Build and test the code, run a smoke test, and run the experiment.
Analyse the results and have them checked by agents that did not produce them.
Write a report in the repository and comment on the source issue with a link.

People approve the design, approve the spend, and review what the smoke test or early exploration shows. Everything else runs without them, and every decision the agent makes on their behalf is logged.

Questions can be about any eval, not only graders: how a scorer should treat a class of output, whether a fix changes scores, whether two models differ on a benchmark, whether a dataset has label errors.

Lifecycle

Stage	Who	Produces	Ends when
0. Intake	agent	`question.md`	the question, the decision and the prior evidence are written down
1. Design	agent	`spec.md`, first `budget` in `experiment.yaml`	the spec passes the contract check
Gate 1: design	person	approval recorded in `experiment.yaml`	the spec and the budget ceiling are approved, or sent back with comments
2. Build	agent	code, tests, `plan.md`	tests pass against mock models, no paid calls yet
3. Smoke and explore	agent	`results/smoke/`, `notes/exploration.md`, cost re-estimate	a small paid run has gone through every arm of the design
Gate 2: checkpoint	person	approval or amendments recorded	the re-estimated budget and the analysis code are approved, and anything surprising has been reviewed. Design changes become numbered amendments to the spec.
4. Run	agent	logs, `run.json`, published data	every planned run is complete or the budget stops it
5. Analyse	agent	`results/<run>/summary.md`, tables, charts	every pre-registered analysis has a result
6. Validate	separate agents	`validation.md`	the checks below pass, or their failures are listed in the report
7. Report	agent	`report.md`, `decision.md`	the report passes the claim check
Gate 3: sign-off	person	approval recorded	the report and the decision record are approved for posting
8. Publish	agent	issue comment, published page	the link is posted on the source issue

Gate 3 exists because a comment on an upstream issue is public and other people act on it.

An agent may stop at any stage and come back, since each stage reads and writes only files in the experiment directory. experiment.yaml records the current stage.

When the agent must stop and ask

Outside the three gates, the agent asks a person when:

spend would pass the approved budget,
the smoke test or a run shows something that would change the question, the hypotheses or the decision rule,
a planned analysis turns out to be impossible or meaningless,
it would need to touch anything outside this repository (open a PR elsewhere, change a shared config).

Anything smaller it decides itself and logs in RULINGS.md, with what the decision costs if wrong. That is how both earlier experiments worked, and the log made the review possible.

The experiment contract

Every experiment lives in experiments/<slug>/ and must contain the following. A probe needs less (see Probes).

File	Required from	Contents
`experiment.yaml`	intake	machine-readable state: slug, title, source links, stage, gates (who approved, when, at which commit), budget ceiling and spend, data prefixes
`question.md`	intake	the question in one sentence, the decision it informs and who makes it, what is already known (with links), what is out of scope
`spec.md`	design	the pre-registration (fields below)
`plan.md`	build	the build plan. Optional for small experiments.
`configs/`	build	every run config, each naming the dataset hash it expects
`RULINGS.md`	build	every decision taken on a person’s behalf
`amendments.md`	after gate 1	numbered design changes after approval, each with its reason and the gate that approved it
`notes/`	any	exploration notes, scratch analysis. Not checked.
`results/<run>/`	run	`run.json`, `summary.md`, charts. No benchmark content.
`MANIFEST.json`	run	every object published to the bucket, with size and SHA-256
`validation.md`	validate	the outcome of each check below
`report.md`	report	the report
`decision.md`	report	the decision record (below)
`results/ledger.jsonl`	smoke	every paid run and every analysis computed
`README.md`	intake	a short index, kept current

uv run python -m eval_experiments.contract check experiments/<slug> checks that the files required for the current stage exist, that the front matter parses, that gates were passed in order, that every config’s dataset hash appears in the manifest, and that the report’s claims have sources (below). CI runs it on every experiment.

Pre-registration fields

spec.md is fixed at gate 1, and its SHA-256 is recorded in experiment.yaml with the approval. Later changes go in amendments.md, so a reader can tell what was planned from what was decided after seeing data. Every result in the report is labelled pre-registered or exploratory. The fields follow AsPredicted and OSF’s templates, adapted for model experiments (see prior-art.md). It has these sections:

Question and decision. The question, and the decision rule: “if X then we recommend A, if Y then B, otherwise we say the evidence does not decide”. Written before any data. Also what is out of scope.
Hypotheses. Each with what result would support it and what would count against it.
Design. Units of analysis, conditions and arms, models and why each was picked, datasets and how they are sampled, number of repeats.
Mapping to practice. For each condition, the real artefacts it stands for (helpers, templates, evals, provider settings), how faithfully it reproduces them, and the code search that will find everything exposed. This is what lets the report’s headline name real things.
Measures. Exact definitions, including how failures and errors are counted.
Analysis plan. The primary comparison or comparisons, the statistics (intervals over the independent unit, paired where possible, how multiple comparisons are handled), and which analyses are exploratory.
Sample size. Why the number of cases is enough for the primary comparison: an expected effect size and the interval width it gives. The smoke run re-estimates the variance and re-checks this at gate 2.
Validity threats. Confounds, and what the design does about each.
Budget. An estimate per model and in total, and a ceiling.

Default statistics, unless the spec argues otherwise, follow Miller, “Adding Error Bars to Evals”: an interval with every rate, clustered on the unit that was sampled (a case, not a reply), paired differences when two conditions see the same cases, and Wilson or Bayesian intervals for small counts.

Report format

report.md is plain markdown with YAML front matter, so it renders on GitHub and Quarto renders it unchanged into the site at https://eval-experiments.generality.org. The front matter generates the evidence card at the top of the note and the row on the front page:

---
title: "Grader prompt structure"
date: 2026-10-16
status: final            # draft | final
kind: experiment         # experiment | probe
question: "How should a judge prompt wrap the text it grades, so the answer cannot steer the judge or the parser?"
decision: "Wrap every field in an unforgeable delimiter, add the warning sentence, keep the instructions after the data, keep reasoning first with an anchored parser."
feeds:
  - label: inspect_evals#2600
    url: https://github.com/UKGovernmentBEIS/inspect_evals/issues/2600
run: "6 judges, 14 prompt variants, 93,940 replies"
spend_usd: 95.41
links:
  spec: spec.md
  results: results/main/summary.md
  rulings: results/RULINGS.md
---

decision, feeds, run and links are omitted while they do not exist, not left blank. The body has no # Title line; the front matter title is the title.

Rules that keep it portable:

Standard markdown only: pipe tables, relative links, and figures as ![caption](results/main/figure.svg). No Quarto shortcodes, callouts or @fig- cross-references, which GitHub shows as raw text.
One paragraph per line, as in the rest of the repository.
Every number in the text links to the table it came from ([0.22](results/main/summary.md#headline)), or sits in a table that is itself generated by the analysis. The claim check enforces this.

Sections, in order: the headline (below), for a reader who stops there; the question and decision; what was done, in short; results, with the primary result first; what the results do not show; method in detail; deviations from the spec; validation; reproduction steps; links to the spec, rulings and data.

The headline

The report opens with a headline written for someone who will act on it, not for someone who wants to know how the experiment worked. The grader-prompt-structure headline is the model: it led with “Inspect’s default model-graded layout is easy to exploit”, not “W1 had the highest attack success rate”. A headline has six parts, in this order:

A claim about a real, named thing. A template, helper, eval, provider or model that people use, named as they know it. Not the experiment’s label for a condition.
How big, in plain words. The worst case and the typical case, per model or setting where they differ, with the measure described (“an answer that writes its own [END DATA] flipped the verdict 90% of the time”), not only named (“ASR 0.90”).
What is exposed. The evals, helpers or users that depend on the thing, found by searching the code and listed by name, with the most exposed one picked out and the reason (“FORTRESS stands out, because it grades responses to adversarial prompts”). Every name links to the code that shows the exposure.
What fixes it, and what the fix costs. Only fixes the experiment measured, with their measured effect and side effects.
The recommendation. Specific actions, and who takes each one.
How far the test matches the real thing. Where the tested condition differs from the real artefact (a copied template with different field names, a judge other than the eval’s default), stated next to the claim, not only in the limitations section.

Parts 3 and 6 need work during design and build, not only at write-up. The spec’s mapping to practice (pre-registration field 4) lists the real artefacts each condition stands for, and the build stage searches the code for everything that uses them.

Decision record

decision.md is a short record in the style of an architecture decision record (MADR): the context, the options, the recommendation, and the evidence behind it, with links to the report, the spec hash, the analysis commit and the logs. It adds the limits of what was tested and the conditions under which the decision should be revisited (“reopen if a provider starts returning refusals as errors”). Gate 3 approves it, and the comment on the source issue is a short version of it. The maintainers still make the decision. The record states what the evidence supports.

A _quarto.yml at the repository root can list every experiments/*/report.md as a page. Building the site and publishing it is a later step and needs nothing extra in the reports.

Probes

Some questions are about how a tool behaves, not about models: what inspect score does with an errored sample, or whether a retry restores a sandbox. They can be answered with mock models, at no cost. Gates 1 and 2 exist to approve spend and to stop a design being tuned to real model outputs, and neither applies when nothing is paid for and every model reply is scripted. Such an experiment can run as a probe.

A probe:

uses only mock models, local sandboxes and other free resources. If it needs a paid call, it is an experiment, and starts at intake;
runs intake, build, run, analyse, validate and report, and skips design, smoke, and gates 1 and 2. Validation by an agent that did not write the probe still applies, recorded in validation.md;
has no pre-registration, and every result in its report is labelled exploratory;
needs experiment.yaml (with kind: probe and only the signoff gate), question.md, README.md, report.md, validation.md and results/<run>/, plus RULINGS.md if it took decisions on a person’s behalf. The rest of the contract table does not apply. The code goes in src/eval_experiments/<slug>/ with tests;
follows the report format, headline included, but has no decision.md. Gate 3 approves the report alone;
records the Inspect version it ran against, since the behaviour it reports can change in any release.

A probe can lead to a full experiment when it shows that the answer depends on models or data. Its report then becomes part of that experiment’s question.md.

Guarding against the agent’s own errors

Agents that run experiments make the same mistakes people do, faster: changing the analysis after seeing the data, running until something is significant, and bugs that make a result look better. Published work on research agents also reports agents gaming their own scoring, and LLM reviewers accepting polished but unsound work (see prior-art.md; several of those sources are preprints). So:

The analysis is fixed before the main run. The analysis code is written during build and run on the smoke data. Gate 2 approves it along with the budget, recording its commit. After that, changes to it are amendments, and the pre-registered numbers are always computed with the approved commit.
Ledger. results/ledger.jsonl gets one line, appended by the shared runner and analysis code, for every paid run and every analysis computed, with its commit and purpose. The report states how many there were. Nothing the agent ran is hidden from the reader.
Clean commits only. The runner refuses to start a paid run from a working tree with uncommitted changes, and records the commit in run.json. Inspect logs already record the commit and a dirty flag.
The commit is the version record. uv.lock pins every dependency exactly, and dependencies taken from git (such as inspect_evals at an unreleased commit) are pinned to a full SHA. A run’s commit therefore fixes every version it used, and a run is reproduced with git checkout <commit> and uv sync --frozen. Experiments share one lockfile, so bumping versions for a new experiment leaves earlier ones reproducible at their own commits. experiment.yaml repeats the versions that matter to the question under pins, for readers, and the contract check fails if they differ from the lockfile at the run commit. An experiment gets its own pyproject.toml and lockfile only when it needs versions that conflict with another experiment running at the same time.
Failures go to a person. When validation fails, the agent records it and stops. It does not get detailed feedback and retry until the checks pass.

Validation

Each check is run by an agent that did not write the code or the analysis, with only the experiment directory, its git history and the bucket as input. Validators read the diffs, the logs and the spec, not only the report.

Reproduce. Fetch the logs named in the manifest, re-run the analysis, and compare every table with the committed one. Any difference fails.
Trace claims. List every number and comparative claim in report.md, and find the table cell or computation each one comes from. Every eval or helper the headline names as exposed must be backed by a code reference at the pinned commit, and the search must be re-run to check none was missed. A claim with no source, or one the source does not support, fails.
Conformance. Compare what was run and analysed with spec.md and amendments.md. Each difference not recorded as an amendment fails.
Adversarial review. Look for confounds, missing controls, over-claiming, analyses chosen after seeing the data, and bugs that would make a result look better than it is. Findings go in validation.md and are answered in the report.
Tests. The experiment’s tests pass, and include at least one end-to-end run against mock models whose expected output is known.
Planted faults. Where cheap, the validator runs the analysis on a copy of the data with a known fault planted (labels shuffled, one condition’s outcomes swapped) and checks that the analysis notices.

The report states which checks passed. A failure does not block the report, but it must be listed in it.

Shared code

The earlier experiments built everything inside grader_comparison. About a third of it is not about graders and moves to eval_experiments.core, with grader_comparison importing from there so the earlier results still reproduce:

Core module	From	What
`core.records`	`records.py`	content-hashed JSONL datasets, stable case ids
`core.replay`	`task.py`	`replay_answer()`: feed a stored output to the scorer without calling the model under test
`core.runner`	`run.py`	a config of models × task variants → `eval_set`, with `run.json`, per-model budgets, resume and the git commit
`core.costs`	`extract.py`	billed cost from logged responses, error classification
`core.stats`	`stats.py`	rates and intervals over cases, paired differences, bootstrap
`core.store`	`store.py`	the bucket and manifests (unchanged)
`contract`	new	the contract check
`report`	new	the claim-tracing helper and table rendering

Each experiment keeps its own package for anything specific to it. The runner should take any Inspect task, not only the grader-comparison one, so an experiment can run a real eval from inspect_evals with different settings.

Agent skills

Each stage is a skill in .claude/skills/, and a driver skill runs them in order and stops at gates:

experiment-intake, experiment-design, experiment-build, experiment-smoke, experiment-run, experiment-analyse, experiment-validate, experiment-report, experiment-publish
run-experiment: the driver. It reads experiment.yaml, runs the next stage, and at a gate writes a short request for review (a comment on the source issue or a message to the requester) and stops.

Gates are recorded in experiment.yaml by a person, or by the agent quoting the person’s approval with a link. Stages after a gate refuse to start until the gate is recorded.

Starting an experiment

At first, by hand: start a session and ask the driver skill to run on a link or a prompt. Later, a label such as experiment on an issue in this repository, or a command in a comment, could start a cloud session. That needs the provider keys and bucket credentials as environment secrets, and a budget ceiling per session.

Rollout

This document, the contract and its check, and experiment.yaml for the two existing experiments (backfilled).
Move the shared code into core, with the earlier experiments’ tests unchanged and passing.
The design and validate skills, then the others.
Dogfood on the empty-responses experiment (inspect_evals#2415), and revise all of the above from what goes wrong.
The Quarto site and the publishing step.

Open questions

Whether the report site is public, and if so how benchmark content is kept out of it. The current rule (no benchmark content in the repository) already covers the reports.
Who may approve gates: only the requester, or anyone at Generality Labs.
Whether upstream issues get a comment at gate 1 as well, so maintainers can object to a design before money is spent.