Rulings made during the grader comparison build and runs

Every decision taken on Matt’s behalf, in order, each with what it costs if wrong. Copied from the SDD ledger (.worktrees/grader-comparison/.superpowers/sdd/2026-10-12-grader-comparison/progress.md) on 2026-10-15.

implementers adjust # type: ignore[...] comments to what mypy actually reports (add, remove or re-code them); mypy must be clean — warn_unused_ignores=true makes plan-supplied ignores possibly wrong — cost if wrong: none beyond comment churn.
if ruff PLR2004 flags a float literal comparison in non-test code, hoist it to a named module constant rather than add a noqa — repo convention is no magic comparisons outside tests — cost if wrong: trivial.
the VIRTUAL_ENV mismatch warning uv prints in the worktree is harmless (uv uses the worktree .venv); implementers ignore it — cost if wrong: none, import path verified.
no reviewer re-dispatch for Task 1’s fix round — the fix only rewrote the commit message; controller verified git diff 9c2b9dfd9 ed26e7374 is empty and the trailer reads Opus 5.5 — cost if wrong: none, the tree is byte-identical to the reviewed one.
every later implementer dispatch states the exact trailer text explicitly, because the haiku implementer wrote its own model name — cost if wrong: none.
run.json steps stays an append-only invocation log (each re-run appends, latest record per (judge, grader) is current; previous dict already takes the latest) — re-run spend is worth recording; fix is documentation plus an explicit test assertion on the history, not de-duplication — cost if wrong: a reader of run.json must take the last record per pair.
the budget cut must survive a crash between a judge’s first and second grader — persist cut_after: true on the triggering step and restore epochs=1 for that judge on resume; add a crash-window test — spec says a judge’s graders not yet run keep the reduced count — cost if wrong: one extra run at full epochs for one judge.
epoch verdict agreement counts only compliant replies, over cases with >=2 compliant epochs — an all-failed judge must not score 1.00 consistency — cost if wrong: none, the old metric was misleading.
“which parser was right” uses only non-provisional labels (label_provenance != original_grade) and per-case rates with intervals via case_rate — spec says intervals over cases and this table comes from phase-two labels — cost if wrong: smaller n in that table.
browse_comp_2233 joins the anchored Q1 table (not the token table) but does not contest cases — Task 13’s findings need #2233’s anchored disagreement rate; the contested trigger is defined over G1-G5 — cost if wrong: repo-row disagreements unlabelled in phase two.
labelled-mode accuracy keeps all labelled cases (uncontested keep provisional labels by design) and the accuracy section states the count per label_provenance — cost if wrong: a reader may over-trust provisional-heavy cells; the counts make it visible.
the token sample draws from all non-errored replies (compliant or not) — unanchored parsers matter most on replies that ignore the format — cost if wrong: a few more phase-two labels.
fix minors 5 (nan reasoning tokens in the cost cell), 6 (stray empty adversarial tag section) and 7 (one 2026 constant) in this round — trivial output bugs — cost if wrong: none.
rename build_dataset.py -> make_dataset.py and test_build_dataset.py -> test_make_dataset.py, and narrow model_roles[“grader”] (ModelConfig | list[ModelConfig]) — the repo mypy exclude regex “build” silently skips any path containing “build”, hiding a real union-attr error; controller updates the plan’s Task 10/11 commands — cost if wrong: one rename.
replace the four integer-reference numeric seeds (0 °C, 365 days, 50 m, 64 squares) with decimal-valued quantities, each with rounding-level yes and near-miss no variants — numeric_close exists to test the tolerance boundary — cost if wrong: none.
fold in the cheap minors this round: swap plain-longest-river for an undisputed fact; replace “The British dollar” distractor; one cheetah variant as the bare synonym “Acinonyx jubatus.”; mutate_bases prints a shortfall like sample_simpleqa — cost if wrong: none.
mutants of a simpleqa base whose label phase two corrects must follow the base (hedge stays no; verbose_correct dropped if the base becomes no) — carried to Task 14 (label.py, controller-run) — cost if wrong: stale labels on a few mutants in accuracy tables.
fix Critical 1 (–judges rewrites run.json judges) by filtering in run() without mutating config, merging judge lists in _load_info, plus a multi-judge multi-invocation test — it would corrupt the contested list and H1/H2 tables in Task 12’s per-judge workflow — cost if wrong: none.
loosen strict to accept emphasis around the label and token and an optional trailing “.” or “!” for G1/G2/G5, but not list markers — decoration does not impair a scorer’s reading and the spec allows bold; a list marker before G1’s final line remains a format deviation — cost if wrong: compliance slightly higher than a byte-exact reading; the anchored parsers still show the raw spread.
SYCOPHANCY_LEADING mirrors the PR’s leading regex (optional short label before the token, same token group, no flags) — compliance must mean “the PR’s leading branch decided” — cost if wrong: none.
export browse_comp’s verdict_found as a replies column pr_verdict_found and use it as browse_comp’s compliance; harness anchored pair stays for the Q1 table — Task 13 needs the silent-default rate — cost if wrong: none.
accuracy uses compliant for parsed/correct (fixes browse_comp’s default “no” counting as parsed) — cost if wrong: none (phase two only).
pass retry_on_error=2 to eval_set and add an error_kind replies column (http_4xx / http_5xx / timeout / other from the error message), shown as sample_error: in the failure-modes table — separates mechanism rejection from transient errors for H2 — cost if wrong: a few retried calls.
minors folded into the same wave: hedge mutation on simpleqa bases only for label-yes bases; G3_NOTE mentions Inspect’s text-tool-call rescue; per-step git_commit and inspect_ai version in run.json; warn when judge_budgets is set but usage() is None; cost heading says per 100 successful calls; agreement and retry cells render n/a not nan.
decline minors — dotenv import (python-dotenv is a direct inspect_ai dependency); README pointing at configs/ and charts (created in Tasks 10 and 13); results.csv question/reference/candidate columns (publication decision deferred until after the first run) — cost if wrong: none.
the deferred-ledger triage stands as the reviewer gave it: only Task 6’s multi-judge test is un-deferred (folded into Critical 1); the rest stay deferred.
accept the fixer’s correction on item 8 — controller confirmed in openai_compatible.py that _resolve_chat_choice runs only when a handler exists, which is only when emulate_tools=True (default False); G3_NOTE now says no rescue applies — cost if wrong: none, verified in source.
keep this plan’s SDD workspace and ledger until Tasks 10-14 finish (they continue the same plan, controller-run); skip finishing-a-development-branch for now because Matt asked to keep the branch local with no merge or PR — cost if wrong: none.
real runs use uv run --with openai (Inspect’s OpenRouter provider needs the openai package, absent from the dev group); README documents it; pyproject untouched to keep the change local — cost if wrong: none. Smoke attempt 1 failed before any call (PrerequisiteError), $0 spent.
swap the 2023 judge to openrouter/openai/gpt-3.5-turbo-0613 (June 2023 snapshot, introduced function calling; served via Azure, passes the guardrail) instead of asking for a guardrail change — same vintage and role, no account change — cost if wrong: Matt may prefer the guardrail change; trivially reversible.
gpt-oss-20b on OpenRouter ignores the CoreWeave host (provider.ignore [CoreWeave]) — CoreWeave returns finish_reason error with an empty message on every forced tool call, while Bedrock, DeepInfra and Groq return proper tool calls (probed 2026-10-13); a host fault, not the model — cost if wrong: none; recorded as a finding.
Gemini 3.7 Flash -> google/gemini-3.7-flash direct — via OpenRouter its strict response_format was not enforced (fenced JSON every time, with or without require_parameters) while Google’s API enforces the same schema (probed 2026-10-13); Opus 5.5 G3 stays a 100% sample error because Anthropic’s own API rejects forced tool_choice for the model (a real finding) — cost if wrong: one more Gemini smoke rerun.
generated answers use openrouter/openai/gpt-3.5-turbo-0613 (plain gpt-3.5-turbo is guardrail-blocked on OpenRouter) and openrouter/meta-llama/llama-3.1-8b-instruct, graded by openai/gpt-5.4 direct — same answerer roles as the plan — cost if wrong: one regeneration (~$5).
a pool that cannot fill its share stays short (no redistribution) but the warning names the pool, with a shortfall test — the real pools fill their shares (generated CORRECT 51 >= 40) and silent redistribution would blur the half/half design — cost if wrong: a slightly short slice, visible in the printout.
reject an empty –exclude-grader string (it would match every row) — cost if wrong: none.
stop batch 1, run the other judges first, then retry llama’s G4/G5 at max_connections 4 (separate config, same log_dir); if still throttled, run llama G4 at 1 epoch — keeps require_parameters so the measurement stays a real mechanism result — cost if wrong: llama G4 has fewer epochs than other cells.
controller verification replaces the scoped re-review for this comment-only fix — ran mypy with and without matplotlib on all 27 files (clean), ruff (clean), full suite with matplotlib (206 passed) — cost if wrong: none. Task 13 (charts code): complete (commits 3b5115c3d..efe3f58ef, review clean)
restored logs/grader_comparison/first_cut/run.json config.graders to the five phase-one graders (backup in logs/grader_comparison/superseded/) — llama G4 ran on pre-13a code and overwrote the list; steps untouched — cost if wrong: none, the data is unchanged.
normalise YAML decision values (True->yes, False->no; only yes/no accepted) and re-validate every updated record — REVIEW.md’s own instruction produced booleans — cost if wrong: none.
a contested mutant’s own panel outcome wins over propagation from its base; when they disagree or the mutant is unresolved, add a review entry naming the conflict — mutant panel votes are evidence about that case — cost if wrong: a few more review entries.
drop review entries for mutants removed from the output (quoted_verdict_line / verbose_correct on a flip) — cost if wrong: none.
pad each case’s votes to the panel size from run.json judges with None — a missing judge must never make a unanimous no — cost if wrong: none.
Matt’s explicit decision may override a hand label (he is the authority), but REVIEW.md states hand entries are informational unless he intends an override; warn on decision ids that are unknown or already settled; REVIEW.md lists a base’s mutants and a mutant’s base — cost if wrong: none.