Empty responses: run plan

October 22, 2026

Empty responses: run plan

The order of work from the end of the build to gate 2, and from gate 2 to the report. Every command runs from the repository root. Paid runs need a clean working tree, the provider keys and bucket credentials in the environment or .env, and network access to the providers’ hosts. In a Claude Code cloud session, which strips ANTHROPIC_API_KEY, the Anthropic key is set as EVAL_EXPERIMENTS_ANTHROPIC_API_KEY and the runner copies it across. The runner writes logs under data/ (not committed) and appends every step to results/ledger.jsonl, which is committed after each run.

Before gate 2

  1. Configs. uv run python -m eval_experiments.empty_responses.make_configs writes every config from configs/_common.yaml. Commit the result.
  2. Exposure search. uv run python -m eval_experiments.empty_responses.exposure --out experiments/empty-responses/notes/exposure.md. Done at build time; re-run if the inspect_evals pin changes.
  3. Exploration. Scan logs we already hold for empty replies, with extract.load_replies on each log directory: grader-parsing’s answer-generation logs (grader-parsing/logs/sources/ in the bucket), and any inspect_evals evaluation-report or Hawk logs available. Write the class rates and a few examples per class to notes/exploration.md.
  4. Smoke run, about $10:
    • uv run python -m eval_experiments.empty_responses.run --config experiments/empty-responses/configs/smoke.yaml
    • uv run python -m eval_experiments.empty_responses.run --config experiments/empty-responses/configs/smoke_part_c.yaml
  5. Analysis on the smoke data: uv run python -m eval_experiments.empty_responses.analyse summarise data/empty-responses/logs/smoke --part-c data/empty-responses/logs/smoke_part_c --config experiments/empty-responses/configs/smoke.yaml --out experiments/empty-responses/results/smoke. Check that the P0 check matches every cell, that each eval’s reply was extracted, and that each route’s settings reached the provider (reasoning effort, max_tokens, Gemini safety settings) by reading one request per route in the logs.
  6. Cost re-estimate from the smoke run’s ledger, per config, against the ceiling.
  7. Gate 2 request: the smoke summary, the exploration, the re-estimate, anything surprising, and the analysis commit to freeze.

After gate 2

  1. Main run: configs/main.yaml. Commit the ledger.
  2. Select re-runs: uv run python -m eval_experiments.empty_responses.analyse select-reruns data/empty-responses/logs/main --out data/empty-responses/reruns, then publish the two id files with the logs.
  3. Re-runs: configs/reruns.yaml.
  4. Part C: configs/part_c.yaml.
  5. Analysis: summarise over main, re-runs and Part C, into results/main/, with the frozen analysis commit.
  6. Publish logs and per-reply tables to the bucket with eval_experiments.store publish, recording them in MANIFEST.json.
  7. Validation by agents that did not write the code (framework section “Validation”), then the report, the decision record and gate 3.