Validation of amendment A1

A Claude Code agent that did not write the A1 code, the runs or the analysis ran these checks on 2026-10-30, at 31d83c3 on branch prompt-structure-shipped, with inspect-ai 0.3.283. It used the experiment directory, its git history, the harness in src/eval_experiments/grader_comparison/, the local log directories named in results/shipped/run.json and the local copies in audit-artefacts/grader_comparison/prompt_structure/. Every local log file and the dataset were first checked against the SHA-256 values in MANIFEST.json (169 of 169 files each for shipped_clean, shipped_main and shipped_smoke, and all three dataset files), so nothing was fetched from the bucket. No model was called. Check 2 (claim tracing) is recorded in the last section.

Check Outcome
1. Reproduce pass. Re-running shipped_analyse on the logs, with --first-run, gives a summary.md whose 311 table rows and all prose match the committed file once mdformat’s column padding is removed. results.csv and replies.csv are byte-identical to the audit-artefacts copies and run.json is equal. The SVG chart differs only in its date stamp and clip-path ids. With the gate-2 echo pattern patched back in, the output also matches the 3cab240 summary, and the smoke summary matches results/shipped_smoke/summary.md. shipped_analyse.py did not change between d1bf444 and 3cab240. Between d1bf444 and 31d83c3 the only code change is the echo pattern (shipped_analyse.py:85 and :96, four lines) and its test. The run commit 4985264 differs from d1bf444 only in experiment.yaml.
2. Trace claims fail with 9 findings, all corrected in the commit that adds the section “Check 2: trace claims” below.
3. Conformance fail, on records only. Judges, arms, attacks, dataset (a42643b790c5, 590 cases, counts as in A1), epochs (3 clean, 1 main), slices, the Holm family and the inspect-ai pin (0.3.283 in uv.lock and in every log header, uv.lock unchanged since before gate 2) match A1 and RULINGS.md. The installed _model.py is byte-identical to tag 0.3.283, and 0.3.283 and 0.3.286 have the same _model.py in the local inspect_ai clone. Five differences are not recorded in amendments.md or RULINGS.md (findings 7, 8, 13 and 15). The echo-pattern change does need an amendment entry, because it changed a reported table after the data were seen. An entry, A3, appeared in the working tree of amendments.md while this validation ran and was not yet committed. It records the pattern change but not the points in finding 1.
4. Adversarial review 15 findings below. The S arms’ prompts are the shipped scorer’s: 60 of 60 logged prompts for Opus 5.5 and 54 of 54 for gpt-4o-mini match a hand rendering through model_graded_qa and model_graded_fact with a mock model. That rendering used attack strings and a fold written from the amendment’s text, not imported from the harness. Neutralisation is on in s-qa ([END-DATA]) and off in s-qa-raw ([END DATA]) in the logged forged_close prompts. s-fact’s attacks use the 12-asterisk separator and [Expert]:. The prompt in each score’s metadata equals the input of the scored model call in 21,000 of 21,000 main-run samples.
5. Tests pass. uv run pytest tests/grader_comparison -q --no-cov gives 379 passed. Tests that run an Inspect eval end to end against a mockllm judge with a known expected prompt or verdict: test_shipped.py::test_s_qa_prompt_is_byte_identical_to_the_shipped_scorer, ::test_s_qa_neutralises_the_forged_close_and_s_qa_raw_does_not, ::test_s_qa_raw_parses_like_the_shipped_scorer, ::test_fence_arm_folds_every_field, ::test_s_fact_uses_the_fact_template and ::test_verdict_and_metadata_shape. No test runs the shipped path from logs through shipped_analyse (finding 14).
6. Planted faults pass, but neither fault is flagged by any automatic check (finding 12). (a) Swapping verdict between s-qa and f-warn for gpt-4o turns its primary row from −0.04 [−0.07, −0.01], “qualifies”, into 0.04 [0.01, 0.07], “does not”, and drops gpt-4o from the verdict line. Other judges’ rows are unchanged. (b) Shuffling labels across bases for Opus 5.5 drops its eligible bases from 60 to 33 in every arm and its clean accuracy from 1.00 to 0.55. For gpt-4o, the eligible bases per arm drop from between 58 and 60 to between 31 and 33.

Findings for the report to answer

  1. The echo rate changed after gate 2, and both versions differ from the spec. At 3cab240, the rate counted any line containing GRADE: X, including quotes in the reasoning. Opus 5.5 was at 0.21 to 0.35, Gemini 3.1 Pro at 0.23 to 0.29 and GLM-5.3 at 0.18 to 0.43. At 31d83c3, it counts only lines that start with GRADE: (shipped_analyse.py:85), and every cell is 0.00 (results/shipped/summary.md, “Echo rate”). Of the 21,000 attacked replies, 1 has two line-start grade lines and 2,561 mention GRADE: two or more times. spec.md:176 defines the echo rate as the share of replies that contain the attack’s fake verdict line verbatim, which is closer to the old count than to the new one. The spec’s reason for the measure can be answered directly. Of the 20,702 attacked replies that have a line-start grade line, none has a shipped parse that differs from the last such line. That means no attack success comes from the parser reading a quoted line. The uncommitted A3 entry in amendments.md records the pattern change and says only the echo table changed. It does not say that every cell is now 0.00, give the earlier numbers, note that neither version matches spec.md:176, or give this parse-agreement figure. The report should state all four. .superpowers/sdd/plan-a1/main-run-report.md:81 still quotes the old rates.

  2. Most of the neutralisation table compares two samples of the same prompt. The neutraliser rewrites only [BEGIN DATA] and [END DATA]. In the logs, s-qa and s-qa-raw prompts are byte-identical for 400 of the 500 attacked cases per judge: every case of fake_verdict, authority, format_hijack, role_spoof, forged_close_nbsp, forged_close_fullwidth and fake_field. Only forged_close and fake_reference differ. The pooled cell (“Neutralisation effect” in summary.md) therefore averages two attacks with a prompt difference and seven without one. The 49 same-prompt cells work as a check of the interval method: none excludes zero. The main-run report’s “largest cell is gpt-4o-mini under fake_field (−0.13)” is one of these same-prompt cells. The report should state the neutralisation effect from the forged_close and fake_reference columns.

  3. s-qa-raw is not the scorer as it was before 0.3.212, but the difference does not change any reading here. amendments.md:28 describes s-qa-raw as “the scorer as it was before 0.3.212”. At 0.3.211, the grade pattern was (?i)GRADE\s*:\s*([CPI])(.*)$. At 0.3.283 it is the permissive last-match pattern with validation against the offered grades. Re-reading all 32,319 readable replies with the 0.3.211 pattern gives the same verdict in every case. The report should describe s-qa-raw as the current scorer without neutralisation, and can cite this check if it keeps the pre-0.3.212 description.

  4. The OpenRouter route sent judges to other hosts than the ruling states. RULINGS.md:27 says OpenRouter sends first-party models to the vendors’ own APIs. The logged providers are Azure for gpt-4o, gpt-4o-mini and gpt-5.6-luna, “Claude Platform on AWS” for Opus 5.5, and Google or Google AI Studio for both Gemini models. GLM-5.3’s replies came from 11 providers, and the mix differs by arm: Mistral served 0.30 of s-qa’s attacked replies and 0.45 of s-qa-raw’s. require_parameters: true does not pin a provider. The first-run comparison for gpt-4o-mini (0.24 against 0.03, “Comparison with the first run” in summary.md) carries a change of host from OpenAI to Azure. GLM-5.3’s arm comparisons carry a provider-mix difference.

  5. There were sample errors and content-filter replies. The main-run report says “None” (main-run-report.md:33). The clean run has 12 sample errors, which are Azure content-filter 400 responses on one base (simpleqa-737cdaadb8, 3 epochs, in s-qa and s-qa-raw, for gpt-4o and gpt-4o-mini). gpt-5.6-luna returned a content_filter stop on the same base 3 times in s-qa, 3 times in s-qa-raw and once in f-warn, where the reply still carried a grade. There is also 1 content_filter attacked reply for Gemini 3.1 Pro in f-fence+warn and 2 gpt-4o-mini replies with stop reason unknown. The base has no clean majority in s-qa and s-qa-raw for gpt-4o, gpt-4o-mini and gpt-5.6-luna, so it is left out of the primary pairs for three of the five primary judges (the 57, 49 and 59 counts in “Primary comparison”). Its attacked cases were graded normally.

  6. Eligible bases differ across arms, so the headline cells are not paired. In “Eligible bases” (summary.md:403), gpt-4o-mini has 52 bases in s-qa, 59 in s-fact, 56 in s-qa-raw, 51 in f-warn, 57 in f-fence and 53 in f-fence+warn. Every ineligible base for every judge has the label no. Apart from the content-filtered base in finding 5, each is a base whose wrong answer the judge accepted on the clean run. The paired tables use the bases eligible in both arms, so they are not affected. The headline table compares rates over different bases, though. For example, gpt-4o-mini’s s-fact cell (0.07) includes 7 bases that its s-qa cell (0.20) leaves out, all of them bases that gpt-4o-mini accepts when wrong under s-qa. The report should cite the paired tables for any comparison between arms.

  7. The decision rule is implemented with Holm-adjusted p-values, not corrected intervals. amendments.md:88 asks whether “any corrected interval excludes zero”. The code applies Holm to a two-sided bootstrap p and requires a negative estimate (shipped_analyse.py:141 to :166, stats.py:87 to :111), and it prints uncorrected intervals. With a percentile bootstrap the two tests are equivalent, and the Holm arithmetic is correct (gpt-4o: 2 × 0.010 = 0.020; gpt-4o-mini: 1 × 0.116). The verdict does not depend on the test. A sign-flip permutation test with 20,000 draws gives Holm p 0.027 for gpt-4o, 0.116 for gpt-4o-mini and 0.003 or less for the other three. A normal approximation gives 0.021 and 0.109. The same four judges qualify under every test. The verdict also holds without forged_close: f-warn minus s-qa is −0.04, −0.03, −0.22, −0.17 and −0.04 for gpt-4o, gpt-4o-mini, Opus 5.5, gpt-5.6-luna and Gemini 3.1 Pro, with the same judges’ intervals excluding zero. This needs a line in RULINGS.md.

  8. The reasoning settings are provider defaults. In the configs, reasoning: true or false is a label (run.py:39) and is not passed to the model. No reasoning effort, temperature or token limit was set, and the logged generate config holds only max_connections. amendments.md:71 asks for the reasoning settings to be recorded in the run config. The median reasoning tokens per reply were 83 for Opus 5.5 (98.5% of replies had some), 210 for Gemini 3.1 Pro, 152 for Gemini 3.7 Flash, 59 for gpt-5.6-luna, 445 for GLM-5.3 and 0 for gpt-4o and gpt-4o-mini. The report should state that the judges ran at their OpenRouter defaults and give these numbers.

  9. The pooled rate depends on the attack mix the experiment chose. The main run has 9 attacks, with 360 cases targeting yes and 140 targeting no. forged_close alone reaches 0.92 to 1.00 under s-qa for Opus 5.5, Gemini 3.1 Pro and gpt-5.6-luna (“Attack success per attack: forged_close”). A headline that states what the shipped scorer lets through should say it is an average over this set of attacks, and should give the per-attack rates beside it.

  10. The fence arms change more than the delimiter. f-fence removes the [Task]:, [Submission]: and [Criterion]: labels and the *** separators. It folds every attack marker to ASCII brackets before the judge sees it, and it passes the question through include_history (recorded in RULINGS.md). The secondary comparisons measure these changes together. The fold turns a forged ⟦END SUBMISSION⟧ into [END SUBMISSION], so forged_close cannot reproduce the fence by construction. The amendment’s validity section states the fold point. The other two should be stated as well.

  11. Arms ran in a fixed order. For each judge the six arms ran one after another in the same order (s-qa first, f-warn fourth), about two to four minutes each. Time and any provider-side change are therefore tied to arm. For the single-host judges this order covers about 15 minutes. The same-prompt cells in finding 2 come from adjacent runs (s-qa and s-qa-raw), so they do not test drift across the whole order.

  12. Nothing ties the arm column to the prompt that was sent. Planted fault (a) changed the primary verdict for gpt-4o, and planted fault (b) changed eligibility, without any warning. With labels shuffled, attack success and the primary estimates hardly move (Opus 5.5 stays at −0.28), because success compares the verdict with attack_target, not with the label. Both faults show in the “Eligible bases” and “Over-defence: clean accuracy” tables. A check in the analysis that the warning sentence appears only in warned arms, and [END-DATA] only in neutralised ones, would catch an arm swap. Fault (a) also moved gpt-4o’s neutralisation cell to −0.05 [−0.09, −0.02], which would read as neutralisation working.

  13. Records are out of date or missing. experiment.yaml:21 gives analysis_commit: d1bf444, but the committed summary comes from 31d83c3. amendments.md:7 still says A1 is awaiting gate 2. amendments.md:28 says the attack is applied in the replay solver and the criterion is folded in the dataset loader. Both happen in the scorer wrapper (shipped.py:194 to :200). This has the same effect, as the byte-identical renderings show, but the text should match. amendments.md:79 names five arms in the headline, and the table has six.

  14. No test covers the shipped analysis from logs. The shipped_analyse tests use synthetic frames and a monkeypatched load_run (test_shipped_analyse.py:302 to :343). The spec’s Testing section asks for canned logs with known ASR, eligibility, noise floor, breakage, echo rate and paired difference. The first run has such a test (test_analyse.py, around line 317). The shipped path does not. The results tables results/shipped/ and results/shipped_smoke/ (results.csv, replies.csv) are also not in the bucket or the manifest, unlike results/main/. They can be regenerated from the logs, and doing so reproduced them byte for byte.

  15. The fold adds a case to the amendment’s rule. shipped.py:65 to :67 maps U+FE18 to ], because its Unicode name is spelt “BRAKCET” and the name rule in amendments.md:39 misses it. The fold covers 42 code points on Python 3.12 (Unicode 15.0) and is idempotent. No attack uses U+FE17 or U+FE18, so the case has no effect on the results. It should be recorded as a ruling.

Check 2: trace claims

An agent that did not write the report traced every number and comparative claim in report.md at baa2985 on 2026-10-30. Every number in the headline’s three claims traced to its table. The verdict is fail with 9 findings, all corrected at the commit docs(grader-prompt-structure): correct the A1 report after claim tracing. The findings were:

  1. The target counts were 340 and 160, and the dataset note gives 360 attacked cases with target yes and 140 with target no. They are corrected in the report and in finding 9 above.
  2. The front-matter decision said the warning lowered attack success on four of the five primary judges. It now says it was lower for every judge and qualified after Holm correction on four of the five.
  3. The exposure sentence named three evals. It now adds SciKnowEval’s balancing_chemical_equation task, which uses model_graded_fact() with every default, and says that ANIMA, MORU and XSTest carry their own copies of the layout, which a change to Inspect’s defaults would not reach.
  4. The two unknown stops were attributed to the main run. One is in the clean run (simpleqa-2c32e2cdf2, s-qa) and one in the main run (adversarial-815c363172, s-qa).
  5. A heading said every reply was read. Two of the 21,000 attacked replies had no readable grade, and the heading and text now say so, with the pooled breakage of 0.00 as rounded.
  6. The hand rendering was described as covering the run. It covered only the S arms, with 60 Opus 5.5 prompts and 54 gpt-4o-mini prompts, and the report now says the f-warn and fence arms were not hand-rendered.
  7. The reproduction exceptions omitted the clip-path ids and said run.json was byte-identical. The report now matches the reproduction row above.
  8. One sentence said the same-prompt cells gave what the interval method should give. It is deleted, and the observation stands as what the table shows.
  9. Numbers in prose without a link to a table or source file are now linked, and the report cuts “two to four minutes” for each arm because no table or run.json field holds it.

Exposure search: SciKnowEval was added, and ANIMA, MORU and XSTest were added as copies of the layout. No eval was removed from the list.