Rulings

Every decision taken on Matt’s behalf during the build of amendment A1, in order, each with what it costs if wrong. Each was made by the Claude Code session on 2026-10-29. Matt approved those marked as approved.

  • The fence arms fold the question through include_history. TaskState.input has no setter on inspect-ai 0.3.283, so the harness cannot rewrite the question before the scorer reads it. The fence arms pass model_graded_qa(include_history=lambda state: fold(state.input_text)), which makes the scorer see the folded question. The S arms and f-warn pass template only. Approved by Matt. Cost if wrong: the fence arms differ from the other arms in one scorer argument, which the report states.

  • The S arms resolve the judge through the default model_role="grader". s-qa-raw calls get_model() instead. Nothing binds the grader role in these runs, so the role falls back to the model under evaluation, which is the judge. Cost if wrong: none while no run binds the role; a run that did would send the S arms to a different model than s-qa-raw.

  • parse_shipped("GRADE: C/I") returns C. This matches the shipped permissive pattern, whose permissive pattern captures the word characters after GRADE:, so GRADE: C/I reads as C, while GRADE: CI and GRADE: $LETTER are parse failures. The amendment’s earlier sentence said it was a parse failure, and that sentence was wrong. Amendment A1 is corrected. Approved by Matt. Cost if wrong: a reply ending GRADE: C/I counts as a correct verdict rather than as breakage.

  • The manifest prefix for the extended dataset uses the 12-character hash. The key is grader-prompt-structure/datasets/a42643b790c5/, as the existing entries use the first 12 characters of the hash. Cost if wrong: none.

  • The analysis warning column for an arm is WARNING in arm.template. The column is true when the arm’s template contains the warning sentence. Cost if wrong: an arm that adds the warning by another route would show False; no arm does.

  • _heatmap gained a two-line change, and _paired_chart is skipped. The heatmap now draws arm ids. The paired chart compares against the W0 arm, which this run does not have, so it is not drawn. Cost if wrong: one chart fewer than the first run.

  • Every judge runs through OpenRouter. Approved by Matt in the Claude Code session on 2026-10-29. One OpenRouter key then serves the whole run. openrouter_usage() and billed_cost() track spend only for OpenRouter calls, so routing every judge through it tracks spend for all seven. The HLE judge comparison ran every judge the same way. The ids change as follows, and every judge carries model_args: {provider: {require_parameters: true}}:

    • openai/gpt-4o becomes openrouter/openai/gpt-4o.
    • openai/gpt-4o-mini becomes openrouter/openai/gpt-4o-mini.
    • openrouter/z-ai/glm-5.3 is unchanged.
    • google/gemini-3.7-flash becomes openrouter/google/gemini-3.7-flash.
    • anthropic/claude-opus-5-5 becomes openrouter/anthropic/claude-opus-5.5, which is OpenRouter’s id for the model.
    • openrouter/openai/gpt-5.6-luna is unchanged.
    • google/gemini-3.1-pro-preview becomes openrouter/google/gemini-3.1-pro-preview.

    The first run reached gpt-4o-mini, Gemini 3.7 Flash and Opus 5.5 through the vendors’ own APIs. require_parameters: true does not pin a host, and the logs show that OpenRouter chose these: Azure for gpt-4o, gpt-4o-mini and gpt-5.6-luna, “Claude Platform on AWS” for Opus 5.5, Google or Google AI Studio for both Gemini models, and eleven providers for GLM 5.3, with a mix that differs by arm (Mistral served 0.30 of s-qa’s attacked replies and 0.45 of s-qa-raw’s). This sentence was corrected after validation (amendment A4); it first said that OpenRouter sends first-party models to the vendors’ own APIs. shipped_analyse.py maps this run’s ids to the first run’s ids for the comparison. Cost if wrong: the one-row comparison with the first run carries a route difference, which its prose states. For gpt-4o-mini that difference includes a change of host from OpenAI to Azure, and GLM 5.3’s comparisons between arms carry a provider-mix difference.

The next two were recorded on 2026-10-30, after validation, for decisions the build had taken without recording them. Amendment A4 gives the detail.

  • The primary comparison is decided by Holm-adjusted bootstrap p-values. shipped_analyse.py applies Holm to a two-sided bootstrap p over the five primary judges and requires a negative estimate. It prints uncorrected intervals, where amendment A1 asked for corrected ones. Cost if wrong: none found. With a percentile bootstrap the two tests agree, and the validator found the same four judges qualify under a permutation test and a normal approximation.

  • The fold maps U+FE18 to ] directly. Its Unicode name is misspelt “BRAKCET”, so the name rule in amendment A1 misses it. Cost if wrong: none for these results, because no attack uses U+FE17 or U+FE18.