Empty responses: analysis tables

Class rates (%)

arm eval route answer error filtered reasoning_only tool_only truncated_empty truncated_text
default air_bench gemini-3.8-flash 0.0 100.0 0.0 0.0 0.0 0.0 0.0
default air_bench gemini-3.8-flash-openrouter 0.0 0.0 80.0 20.0 0.0 0.0 0.0
default air_bench gpt-4.1 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default air_bench gpt-5.4-mini 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default air_bench sonnet-5 0.0 60.0 40.0 0.0 0.0 0.0 0.0
default air_bench sonnet-5-openrouter 0.0 100.0 0.0 0.0 0.0 0.0 0.0
default bfcl_irrelevance gemini-3.8-flash 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default bfcl_irrelevance gemini-3.8-flash-openrouter 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default bfcl_irrelevance gpt-4.1 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default bfcl_irrelevance gpt-5.4-mini 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default bfcl_irrelevance sonnet-5 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default bfcl_simple_python gemini-3.8-flash 0.0 0.0 0.0 0.0 100.0 0.0 0.0
default bfcl_simple_python gemini-3.8-flash-openrouter 0.0 0.0 0.0 0.0 100.0 0.0 0.0
default bfcl_simple_python gpt-4.1 0.0 0.0 0.0 0.0 100.0 0.0 0.0
default bfcl_simple_python gpt-5.4-mini 20.0 0.0 0.0 0.0 80.0 0.0 0.0
default bfcl_simple_python sonnet-5 20.0 0.0 0.0 0.0 80.0 0.0 0.0
default coconot gemini-3.8-flash 40.0 0.0 0.0 0.0 0.0 0.0 60.0
default coconot gemini-3.8-flash-openrouter 20.0 0.0 0.0 0.0 0.0 0.0 80.0
default coconot gpt-4.1 80.0 0.0 0.0 0.0 0.0 0.0 20.0
default coconot gpt-5.4-mini 40.0 0.0 0.0 0.0 0.0 20.0 40.0
default coconot sonnet-5 0.0 0.0 0.0 0.0 0.0 0.0 100.0
default coconot sonnet-5-openrouter 0.0 0.0 0.0 0.0 0.0 0.0 100.0
default fortress_adversarial gemini-3.8-flash 20.0 0.0 80.0 0.0 0.0 0.0 0.0
default fortress_adversarial gemini-3.8-flash-openrouter 40.0 0.0 60.0 0.0 0.0 0.0 0.0
default fortress_adversarial gpt-4.1 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default fortress_adversarial gpt-5.4-mini 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default fortress_adversarial sonnet-5 80.0 0.0 20.0 0.0 0.0 0.0 0.0
default fortress_adversarial sonnet-5-openrouter 80.0 0.0 20.0 0.0 0.0 0.0 0.0
default fortress_benign gemini-3.8-flash 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default fortress_benign gemini-3.8-flash-openrouter 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default fortress_benign gpt-4.1 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default fortress_benign gpt-5.4-mini 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default fortress_benign sonnet-5 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default fortress_benign sonnet-5-openrouter 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default gpqa_diamond gemini-3.8-flash 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default gpqa_diamond gemini-3.8-flash-openrouter 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default gpqa_diamond gpt-4.1 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default gpqa_diamond gpt-5.4-mini 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default gpqa_diamond sonnet-5 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default hle gemini-3.8-flash 80.0 0.0 20.0 0.0 0.0 0.0 0.0
default hle gemini-3.8-flash-openrouter 60.0 0.0 40.0 0.0 0.0 0.0 0.0
default hle gpt-4.1 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default hle gpt-5.4-mini 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default hle sonnet-5 60.0 0.0 20.0 0.0 0.0 20.0 0.0
default strong_reject gemini-3.8-flash 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default strong_reject gemini-3.8-flash-openrouter 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default strong_reject gpt-4.1 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default strong_reject gpt-5.4-mini 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default strong_reject sonnet-5 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default strong_reject sonnet-5-openrouter 100.0 0.0 0.0 0.0 0.0 0.0 0.0
default xstest_safe gemini-3.8-flash 0.0 0.0 0.0 0.0 0.0 0.0 100.0
default xstest_safe gemini-3.8-flash-openrouter 0.0 0.0 0.0 0.0 0.0 0.0 100.0
default xstest_safe gpt-4.1 0.0 0.0 0.0 0.0 0.0 0.0 100.0
default xstest_safe gpt-5.4-mini 0.0 0.0 0.0 0.0 0.0 20.0 80.0
default xstest_safe sonnet-5 0.0 0.0 0.0 0.0 0.0 0.0 100.0
default xstest_safe sonnet-5-openrouter 0.0 0.0 0.0 0.0 0.0 0.0 100.0
default xstest_unsafe gemini-3.8-flash 60.0 0.0 0.0 0.0 0.0 0.0 40.0
default xstest_unsafe gemini-3.8-flash-openrouter 60.0 0.0 0.0 0.0 0.0 0.0 40.0
default xstest_unsafe gpt-4.1 60.0 0.0 0.0 0.0 0.0 0.0 40.0
default xstest_unsafe gpt-5.4-mini 20.0 0.0 0.0 0.0 0.0 0.0 80.0
default xstest_unsafe sonnet-5 40.0 0.0 0.0 0.0 0.0 0.0 60.0
default xstest_unsafe sonnet-5-openrouter 40.0 0.0 0.0 0.0 0.0 0.0 60.0
tight hle gemini-3.8-flash 60.0 0.0 0.0 0.0 0.0 0.0 40.0
tight hle gemini-3.8-flash-openrouter 60.0 0.0 0.0 0.0 0.0 0.0 40.0
tight hle gpt-5.4-mini 20.0 0.0 0.0 0.0 0.0 80.0 0.0
tight hle sonnet-5 40.0 0.0 20.0 0.0 0.0 40.0 0.0

D1(a): truncated_empty share of all samples, reasoning routes, default arm (%)

eval route max_tokens n rate low high
air_bench gemini-3.8-flash 5 0.0 0.0 43.4
air_bench gemini-3.8-flash-openrouter 5 0.0 0.0 43.4
air_bench gpt-5.4-mini 5 0.0 0.0 43.4
air_bench sonnet-5 42000.0 5 0.0 0.0 43.4
air_bench sonnet-5-openrouter 5 0.0 0.0 43.4
bfcl_irrelevance gemini-3.8-flash 5 0.0 0.0 43.4
bfcl_irrelevance gemini-3.8-flash-openrouter 5 0.0 0.0 43.4
bfcl_irrelevance gpt-5.4-mini 5 0.0 0.0 43.4
bfcl_irrelevance sonnet-5 42000.0 5 0.0 0.0 43.4
bfcl_simple_python gemini-3.8-flash 5 0.0 0.0 43.4
bfcl_simple_python gemini-3.8-flash-openrouter 5 0.0 0.0 43.4
bfcl_simple_python gpt-5.4-mini 5 0.0 0.0 43.4
bfcl_simple_python sonnet-5 42000.0 5 0.0 0.0 43.4
coconot gemini-3.8-flash 256.0 5 0.0 0.0 43.4
coconot gemini-3.8-flash-openrouter 256.0 5 0.0 0.0 43.4
coconot gpt-5.4-mini 256.0 5 20.0 3.6 62.4
coconot sonnet-5 256.0 5 0.0 0.0 43.4
coconot sonnet-5-openrouter 256.0 5 0.0 0.0 43.4
fortress_adversarial gemini-3.8-flash 5 0.0 0.0 43.4
fortress_adversarial gemini-3.8-flash-openrouter 5 0.0 0.0 43.4
fortress_adversarial gpt-5.4-mini 5 0.0 0.0 43.4
fortress_adversarial sonnet-5 42000.0 5 0.0 0.0 43.4
fortress_adversarial sonnet-5-openrouter 5 0.0 0.0 43.4
fortress_benign gemini-3.8-flash 5 0.0 0.0 43.4
fortress_benign gemini-3.8-flash-openrouter 5 0.0 0.0 43.4
fortress_benign gpt-5.4-mini 5 0.0 0.0 43.4
fortress_benign sonnet-5 42000.0 5 0.0 0.0 43.4
fortress_benign sonnet-5-openrouter 5 0.0 0.0 43.4
gpqa_diamond gemini-3.8-flash 5 0.0 0.0 43.4
gpqa_diamond gemini-3.8-flash-openrouter 5 0.0 0.0 43.4
gpqa_diamond gpt-5.4-mini 5 0.0 0.0 43.4
gpqa_diamond sonnet-5 42000.0 5 0.0 0.0 43.4
hle gemini-3.8-flash 5 0.0 0.0 43.4
hle gemini-3.8-flash-openrouter 5 0.0 0.0 43.4
hle gpt-5.4-mini 5 0.0 0.0 43.4
hle sonnet-5 42000.0 5 20.0 3.6 62.4
strong_reject gemini-3.8-flash 2048.0 5 0.0 0.0 43.4
strong_reject gemini-3.8-flash-openrouter 2048.0 5 0.0 0.0 43.4
strong_reject gpt-5.4-mini 2048.0 5 0.0 0.0 43.4
strong_reject sonnet-5 2048.0 5 0.0 0.0 43.4
strong_reject sonnet-5-openrouter 2048.0 5 0.0 0.0 43.4
xstest_safe gemini-3.8-flash 256.0 5 0.0 0.0 43.4
xstest_safe gemini-3.8-flash-openrouter 256.0 5 0.0 0.0 43.4
xstest_safe gpt-5.4-mini 256.0 5 20.0 3.6 62.4
xstest_safe sonnet-5 256.0 5 0.0 0.0 43.4
xstest_safe sonnet-5-openrouter 256.0 5 0.0 0.0 43.4
xstest_unsafe gemini-3.8-flash 256.0 5 0.0 0.0 43.4
xstest_unsafe gemini-3.8-flash-openrouter 256.0 5 0.0 0.0 43.4
xstest_unsafe gpt-5.4-mini 256.0 5 0.0 0.0 43.4
xstest_unsafe sonnet-5 256.0 5 0.0 0.0 43.4
xstest_unsafe sonnet-5-openrouter 256.0 5 0.0 0.0 43.4

D1(a), HLE tight arm (exploratory)

eval route max_tokens n rate low high
hle gemini-3.8-flash 2048.0 5 0.0 0.0 43.4
hle gemini-3.8-flash-openrouter 2048.0 5 0.0 0.0 43.4
hle gpt-5.4-mini 2048.0 5 80.0 37.6 96.4
hle sonnet-5 2048.0 5 40.0 11.8 76.9

D2: refusal form disagreement between routes (%)

route second_route reading paired_samples n rate low high
sonnet-5 sonnet-5-openrouter either 30 16 18.8 6.6 43.0
sonnet-5 sonnet-5-openrouter both 30 13 0.0 0.0 22.8
gemini-3.8-flash gemini-3.8-flash-openrouter either 25 18 27.8 12.5 50.9
gemini-3.8-flash gemini-3.8-flash-openrouter both 25 13 0.0 0.0 22.8

D3: headline under each policy (primary cells marked)

eval route arm headline policy value half_width scored rerouted change p0_half_width primary threshold meets
air_bench gemini-3.8-flash default all (refusal rate) P0 0 0 True 1.0 False
air_bench gemini-3.8-flash default all (refusal rate) P1 0 0 True 1.0 False
air_bench gemini-3.8-flash default all (refusal rate) P2 0 0 True 1.0 False
air_bench gemini-3.8-flash default all (refusal rate) P3 0 0 True 1.0 False
air_bench gemini-3.8-flash-openrouter default all (refusal rate) P0 0.0 0.0 5 0 0.0 0.0 True 1.0 False
air_bench gemini-3.8-flash-openrouter default all (refusal rate) P1 0.0 0.0 5 0 0.0 0.0 True 1.0 False
air_bench gemini-3.8-flash-openrouter default all (refusal rate) P2 0 5 0.0 True 1.0 False
air_bench gemini-3.8-flash-openrouter default all (refusal rate) P3 80.0 39.2 5 4 80.0 0.0 True 1.0 True
air_bench gpt-4.1 default all (refusal rate) P0 100.0 0.0 5 0 0.0 0.0 True 1.0 False
air_bench gpt-4.1 default all (refusal rate) P1 100.0 0.0 5 0 0.0 0.0 True 1.0 False
air_bench gpt-4.1 default all (refusal rate) P2 100.0 0.0 5 0 0.0 0.0 True 1.0 False
air_bench gpt-4.1 default all (refusal rate) P3 100.0 0.0 5 0 0.0 0.0 True 1.0 False
air_bench gpt-5.4-mini default all (refusal rate) P0 100.0 0.0 5 0 0.0 0.0 True 1.0 False
air_bench gpt-5.4-mini default all (refusal rate) P1 100.0 0.0 5 0 0.0 0.0 True 1.0 False
air_bench gpt-5.4-mini default all (refusal rate) P2 100.0 0.0 5 0 0.0 0.0 True 1.0 False
air_bench gpt-5.4-mini default all (refusal rate) P3 100.0 0.0 5 0 0.0 0.0 True 1.0 False
air_bench sonnet-5 default all (refusal rate) P0 0.0 0.0 2 0 0.0 0.0 True 1.0 False
air_bench sonnet-5 default all (refusal rate) P1 0.0 0.0 2 0 0.0 0.0 True 1.0 False
air_bench sonnet-5 default all (refusal rate) P2 0 2 0.0 True 1.0 False
air_bench sonnet-5 default all (refusal rate) P3 100.0 0.0 2 2 100.0 0.0 True 1.0 True
air_bench sonnet-5-openrouter default all (refusal rate) P0 0 0 True 1.0 False
air_bench sonnet-5-openrouter default all (refusal rate) P1 0 0 True 1.0 False
air_bench sonnet-5-openrouter default all (refusal rate) P2 0 0 True 1.0 False
air_bench sonnet-5-openrouter default all (refusal rate) P3 0 0 True 1.0 False
bfcl_irrelevance gemini-3.8-flash default irrelevance_acc P0 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_irrelevance gemini-3.8-flash default irrelevance_acc P1 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_irrelevance gemini-3.8-flash default irrelevance_acc P2 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_irrelevance gemini-3.8-flash default irrelevance_acc P3 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_irrelevance gemini-3.8-flash-openrouter default irrelevance_acc P0 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_irrelevance gemini-3.8-flash-openrouter default irrelevance_acc P1 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_irrelevance gemini-3.8-flash-openrouter default irrelevance_acc P2 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_irrelevance gemini-3.8-flash-openrouter default irrelevance_acc P3 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_irrelevance gpt-4.1 default irrelevance_acc P0 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_irrelevance gpt-4.1 default irrelevance_acc P1 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_irrelevance gpt-4.1 default irrelevance_acc P2 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_irrelevance gpt-4.1 default irrelevance_acc P3 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_irrelevance gpt-5.4-mini default irrelevance_acc P0 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_irrelevance gpt-5.4-mini default irrelevance_acc P1 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_irrelevance gpt-5.4-mini default irrelevance_acc P2 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_irrelevance gpt-5.4-mini default irrelevance_acc P3 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_irrelevance sonnet-5 default irrelevance_acc P0 60.0 48.0 5 0 0.0 48.0 False 48.0 False
bfcl_irrelevance sonnet-5 default irrelevance_acc P1 60.0 48.0 5 0 0.0 48.0 False 48.0 False
bfcl_irrelevance sonnet-5 default irrelevance_acc P2 60.0 48.0 5 0 0.0 48.0 False 48.0 False
bfcl_irrelevance sonnet-5 default irrelevance_acc P3 60.0 48.0 5 0 0.0 48.0 False 48.0 False
bfcl_simple_python gemini-3.8-flash default simple_python_acc P0 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_simple_python gemini-3.8-flash default simple_python_acc P1 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_simple_python gemini-3.8-flash default simple_python_acc P2 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_simple_python gemini-3.8-flash default simple_python_acc P3 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_simple_python gemini-3.8-flash-openrouter default simple_python_acc P0 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_simple_python gemini-3.8-flash-openrouter default simple_python_acc P1 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_simple_python gemini-3.8-flash-openrouter default simple_python_acc P2 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_simple_python gemini-3.8-flash-openrouter default simple_python_acc P3 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_simple_python gpt-4.1 default simple_python_acc P0 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_simple_python gpt-4.1 default simple_python_acc P1 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_simple_python gpt-4.1 default simple_python_acc P2 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_simple_python gpt-4.1 default simple_python_acc P3 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_simple_python gpt-5.4-mini default simple_python_acc P0 80.0 39.2 5 0 0.0 39.2 False 39.2 False
bfcl_simple_python gpt-5.4-mini default simple_python_acc P1 80.0 39.2 5 0 0.0 39.2 False 39.2 False
bfcl_simple_python gpt-5.4-mini default simple_python_acc P2 80.0 39.2 5 0 0.0 39.2 False 39.2 False
bfcl_simple_python gpt-5.4-mini default simple_python_acc P3 80.0 39.2 5 0 0.0 39.2 False 39.2 False
bfcl_simple_python sonnet-5 default simple_python_acc P0 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_simple_python sonnet-5 default simple_python_acc P1 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_simple_python sonnet-5 default simple_python_acc P2 100.0 0.0 5 0 0.0 0.0 False 1.0 False
bfcl_simple_python sonnet-5 default simple_python_acc P3 100.0 0.0 5 0 0.0 0.0 False 1.0 False
coconot gemini-3.8-flash default compliance_rate total P0 0.0 0.0 5 0 0.0 0.0 False 1.0 False
coconot gemini-3.8-flash default compliance_rate total P1 0.0 0.0 5 0 0.0 0.0 False 1.0 False
coconot gemini-3.8-flash default compliance_rate total P2 0.0 0.0 5 0 0.0 0.0 False 1.0 False
coconot gemini-3.8-flash default compliance_rate total P3 0.0 0.0 5 0 0.0 0.0 False 1.0 False
coconot gemini-3.8-flash-openrouter default compliance_rate total P0 20.0 39.2 5 0 0.0 39.2 False 39.2 False
coconot gemini-3.8-flash-openrouter default compliance_rate total P1 20.0 39.2 5 0 0.0 39.2 False 39.2 False
coconot gemini-3.8-flash-openrouter default compliance_rate total P2 20.0 39.2 5 0 0.0 39.2 False 39.2 False
coconot gemini-3.8-flash-openrouter default compliance_rate total P3 20.0 39.2 5 0 0.0 39.2 False 39.2 False
coconot gpt-4.1 default compliance_rate total P0 0.0 0.0 5 0 0.0 0.0 False 1.0 False
coconot gpt-4.1 default compliance_rate total P1 0.0 0.0 5 0 0.0 0.0 False 1.0 False
coconot gpt-4.1 default compliance_rate total P2 0.0 0.0 5 0 0.0 0.0 False 1.0 False
coconot gpt-4.1 default compliance_rate total P3 0.0 0.0 5 0 0.0 0.0 False 1.0 False
coconot gpt-5.4-mini default compliance_rate total P0 0.0 0.0 5 0 0.0 0.0 False 1.0 False
coconot gpt-5.4-mini default compliance_rate total P1 20.0 39.2 5 1 20.0 0.0 False 1.0 False
coconot gpt-5.4-mini default compliance_rate total P2 0.0 0.0 4 1 0.0 0.0 False 1.0 False
coconot gpt-5.4-mini default compliance_rate total P3 0.0 0.0 4 1 0.0 0.0 False 1.0 False
coconot sonnet-5 default compliance_rate total P0 0.0 0.0 5 0 0.0 0.0 False 1.0 False
coconot sonnet-5 default compliance_rate total P1 0.0 0.0 5 0 0.0 0.0 False 1.0 False
coconot sonnet-5 default compliance_rate total P2 0.0 0.0 5 0 0.0 0.0 False 1.0 False
coconot sonnet-5 default compliance_rate total P3 0.0 0.0 5 0 0.0 0.0 False 1.0 False
coconot sonnet-5-openrouter default compliance_rate total P0 0.0 0.0 5 0 0.0 0.0 False 1.0 False
coconot sonnet-5-openrouter default compliance_rate total P1 0.0 0.0 5 0 0.0 0.0 False 1.0 False
coconot sonnet-5-openrouter default compliance_rate total P2 0.0 0.0 5 0 0.0 0.0 False 1.0 False
coconot sonnet-5-openrouter default compliance_rate total P3 0.0 0.0 5 0 0.0 0.0 False 1.0 False
fortress_adversarial gemini-3.8-flash default ARS overall P0 0.0 0.0 5 0 0.0 0.0 False 1.0 False
fortress_adversarial gemini-3.8-flash default ARS overall P1 80.0 39.2 5 4 80.0 0.0 False 1.0 False
fortress_adversarial gemini-3.8-flash default ARS overall P2 0.0 1 4 0.0 0.0 False 1.0 False
fortress_adversarial gemini-3.8-flash default ARS overall P3 0.0 0.0 5 0 0.0 0.0 False 1.0 False
fortress_adversarial gemini-3.8-flash-openrouter default ARS overall P0 0.0 0.0 5 0 0.0 0.0 False 1.0 False
fortress_adversarial gemini-3.8-flash-openrouter default ARS overall P1 60.0 48.0 5 3 60.0 0.0 False 1.0 False
fortress_adversarial gemini-3.8-flash-openrouter default ARS overall P2 0.0 0.0 2 3 0.0 0.0 False 1.0 False
fortress_adversarial gemini-3.8-flash-openrouter default ARS overall P3 0.0 0.0 5 0 0.0 0.0 False 1.0 False
fortress_adversarial gpt-4.1 default ARS overall P0 57.1 23.4 5 0 0.0 23.4 False 23.4 False
fortress_adversarial gpt-4.1 default ARS overall P1 57.1 23.4 5 0 0.0 23.4 False 23.4 False
fortress_adversarial gpt-4.1 default ARS overall P2 57.1 23.4 5 0 0.0 23.4 False 23.4 False
fortress_adversarial gpt-4.1 default ARS overall P3 57.1 23.4 5 0 0.0 23.4 False 23.4 False
fortress_adversarial gpt-5.4-mini default ARS overall P0 11.4 13.7 5 0 0.0 13.7 False 13.7 False
fortress_adversarial gpt-5.4-mini default ARS overall P1 11.4 13.7 5 0 0.0 13.7 False 13.7 False
fortress_adversarial gpt-5.4-mini default ARS overall P2 11.4 13.7 5 0 0.0 13.7 False 13.7 False
fortress_adversarial gpt-5.4-mini default ARS overall P3 11.4 13.7 5 0 0.0 13.7 False 13.7 False
fortress_adversarial sonnet-5 default ARS overall P0 5.7 11.2 5 0 0.0 11.2 False 11.2 False
fortress_adversarial sonnet-5 default ARS overall P1 25.7 38.0 5 1 20.0 11.2 False 11.2 False
fortress_adversarial sonnet-5 default ARS overall P2 7.1 14.0 4 1 1.4 11.2 False 11.2 False
fortress_adversarial sonnet-5 default ARS overall P3 5.7 11.2 5 0 0.0 11.2 False 11.2 False
fortress_adversarial sonnet-5-openrouter default ARS overall P0 2.9 5.6 5 0 0.0 5.6 False 5.6 False
fortress_adversarial sonnet-5-openrouter default ARS overall P1 22.9 38.2 5 1 20.0 5.6 False 5.6 False
fortress_adversarial sonnet-5-openrouter default ARS overall P2 3.6 7.0 4 1 0.7 5.6 False 5.6 False
fortress_adversarial sonnet-5-openrouter default ARS overall P3 2.9 5.6 5 0 0.0 5.6 False 5.6 False
fortress_benign gemini-3.8-flash default ORS overall P0 20.0 39.2 5 0 0.0 39.2 True 39.2 False
fortress_benign gemini-3.8-flash default ORS overall P1 20.0 39.2 5 0 0.0 39.2 True 39.2 False
fortress_benign gemini-3.8-flash default ORS overall P2 20.0 39.2 5 0 0.0 39.2 True 39.2 False
fortress_benign gemini-3.8-flash default ORS overall P3 20.0 39.2 5 0 0.0 39.2 True 39.2 False
fortress_benign gemini-3.8-flash-openrouter default ORS overall P0 40.0 48.0 5 0 0.0 48.0 True 48.0 False
fortress_benign gemini-3.8-flash-openrouter default ORS overall P1 40.0 48.0 5 0 0.0 48.0 True 48.0 False
fortress_benign gemini-3.8-flash-openrouter default ORS overall P2 40.0 48.0 5 0 0.0 48.0 True 48.0 False
fortress_benign gemini-3.8-flash-openrouter default ORS overall P3 40.0 48.0 5 0 0.0 48.0 True 48.0 False
fortress_benign gpt-4.1 default ORS overall P0 20.0 39.2 5 0 0.0 39.2 True 39.2 False
fortress_benign gpt-4.1 default ORS overall P1 20.0 39.2 5 0 0.0 39.2 True 39.2 False
fortress_benign gpt-4.1 default ORS overall P2 20.0 39.2 5 0 0.0 39.2 True 39.2 False
fortress_benign gpt-4.1 default ORS overall P3 20.0 39.2 5 0 0.0 39.2 True 39.2 False
fortress_benign gpt-5.4-mini default ORS overall P0 0.0 0.0 5 0 0.0 0.0 True 1.0 False
fortress_benign gpt-5.4-mini default ORS overall P1 0.0 0.0 5 0 0.0 0.0 True 1.0 False
fortress_benign gpt-5.4-mini default ORS overall P2 0.0 0.0 5 0 0.0 0.0 True 1.0 False
fortress_benign gpt-5.4-mini default ORS overall P3 0.0 0.0 5 0 0.0 0.0 True 1.0 False
fortress_benign sonnet-5 default ORS overall P0 0.0 0.0 5 0 0.0 0.0 True 1.0 False
fortress_benign sonnet-5 default ORS overall P1 0.0 0.0 5 0 0.0 0.0 True 1.0 False
fortress_benign sonnet-5 default ORS overall P2 0.0 0.0 5 0 0.0 0.0 True 1.0 False
fortress_benign sonnet-5 default ORS overall P3 0.0 0.0 5 0 0.0 0.0 True 1.0 False
fortress_benign sonnet-5-openrouter default ORS overall P0 0.0 0.0 5 0 0.0 0.0 True 1.0 False
fortress_benign sonnet-5-openrouter default ORS overall P1 0.0 0.0 5 0 0.0 0.0 True 1.0 False
fortress_benign sonnet-5-openrouter default ORS overall P2 0.0 0.0 5 0 0.0 0.0 True 1.0 False
fortress_benign sonnet-5-openrouter default ORS overall P3 0.0 0.0 5 0 0.0 0.0 True 1.0 False
gpqa_diamond gemini-3.8-flash default accuracy P0 100.0 0.0 5 0 0.0 0.0 False 1.0 False
gpqa_diamond gemini-3.8-flash default accuracy P1 100.0 0.0 5 0 0.0 0.0 False 1.0 False
gpqa_diamond gemini-3.8-flash default accuracy P2 100.0 0.0 5 0 0.0 0.0 False 1.0 False
gpqa_diamond gemini-3.8-flash default accuracy P3 100.0 0.0 5 0 0.0 0.0 False 1.0 False
gpqa_diamond gemini-3.8-flash-openrouter default accuracy P0 100.0 0.0 5 0 0.0 0.0 False 1.0 False
gpqa_diamond gemini-3.8-flash-openrouter default accuracy P1 100.0 0.0 5 0 0.0 0.0 False 1.0 False
gpqa_diamond gemini-3.8-flash-openrouter default accuracy P2 100.0 0.0 5 0 0.0 0.0 False 1.0 False
gpqa_diamond gemini-3.8-flash-openrouter default accuracy P3 100.0 0.0 5 0 0.0 0.0 False 1.0 False
gpqa_diamond gpt-4.1 default accuracy P0 60.0 48.0 5 0 0.0 48.0 False 48.0 False
gpqa_diamond gpt-4.1 default accuracy P1 60.0 48.0 5 0 0.0 48.0 False 48.0 False
gpqa_diamond gpt-4.1 default accuracy P2 60.0 48.0 5 0 0.0 48.0 False 48.0 False
gpqa_diamond gpt-4.1 default accuracy P3 60.0 48.0 5 0 0.0 48.0 False 48.0 False
gpqa_diamond gpt-5.4-mini default accuracy P0 80.0 39.2 5 0 0.0 39.2 False 39.2 False
gpqa_diamond gpt-5.4-mini default accuracy P1 80.0 39.2 5 0 0.0 39.2 False 39.2 False
gpqa_diamond gpt-5.4-mini default accuracy P2 80.0 39.2 5 0 0.0 39.2 False 39.2 False
gpqa_diamond gpt-5.4-mini default accuracy P3 80.0 39.2 5 0 0.0 39.2 False 39.2 False
gpqa_diamond sonnet-5 default accuracy P0 100.0 0.0 5 0 0.0 0.0 False 1.0 False
gpqa_diamond sonnet-5 default accuracy P1 100.0 0.0 5 0 0.0 0.0 False 1.0 False
gpqa_diamond sonnet-5 default accuracy P2 100.0 0.0 5 0 0.0 0.0 False 1.0 False
gpqa_diamond sonnet-5 default accuracy P3 100.0 0.0 5 0 0.0 0.0 False 1.0 False
hle gemini-3.8-flash default hle/accuracy P0 80.0 39.2 5 0 0.0 39.2 False 39.2 False
hle gemini-3.8-flash default hle/accuracy P1 80.0 39.2 5 0 0.0 39.2 False 39.2 False
hle gemini-3.8-flash default hle/accuracy P2 100.0 0.0 4 1 20.0 39.2 False 39.2 False
hle gemini-3.8-flash default hle/accuracy P3 80.0 39.2 5 0 0.0 39.2 False 39.2 False
hle gemini-3.8-flash-openrouter default hle/accuracy P0 60.0 48.0 5 0 0.0 48.0 False 48.0 False
hle gemini-3.8-flash-openrouter default hle/accuracy P1 60.0 48.0 5 0 0.0 48.0 False 48.0 False
hle gemini-3.8-flash-openrouter default hle/accuracy P2 100.0 0.0 3 2 40.0 48.0 False 48.0 False
hle gemini-3.8-flash-openrouter default hle/accuracy P3 60.0 48.0 5 0 0.0 48.0 False 48.0 False
hle gpt-4.1 default hle/accuracy P0 0.0 0.0 5 0 0.0 0.0 False 1.0 False
hle gpt-4.1 default hle/accuracy P1 0.0 0.0 5 0 0.0 0.0 False 1.0 False
hle gpt-4.1 default hle/accuracy P2 0.0 0.0 5 0 0.0 0.0 False 1.0 False
hle gpt-4.1 default hle/accuracy P3 0.0 0.0 5 0 0.0 0.0 False 1.0 False
hle gpt-5.4-mini default hle/accuracy P0 40.0 48.0 5 0 0.0 48.0 False 48.0 False
hle gpt-5.4-mini default hle/accuracy P1 40.0 48.0 5 0 0.0 48.0 False 48.0 False
hle gpt-5.4-mini default hle/accuracy P2 40.0 48.0 5 0 0.0 48.0 False 48.0 False
hle gpt-5.4-mini default hle/accuracy P3 40.0 48.0 5 0 0.0 48.0 False 48.0 False
hle sonnet-5 default hle/accuracy P0 60.0 48.0 5 0 0.0 48.0 False 48.0 False
hle sonnet-5 default hle/accuracy P1 60.0 48.0 5 0 0.0 48.0 False 48.0 False
hle sonnet-5 default hle/accuracy P2 100.0 0.0 3 2 40.0 48.0 False 48.0 False
hle sonnet-5 default hle/accuracy P3 75.0 49.0 4 1 15.0 48.0 False 48.0 False
strong_reject gemini-3.8-flash default jailbreak_rate P0 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject gemini-3.8-flash default jailbreak_rate P1 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject gemini-3.8-flash default jailbreak_rate P2 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject gemini-3.8-flash default jailbreak_rate P3 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject gemini-3.8-flash-openrouter default jailbreak_rate P0 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject gemini-3.8-flash-openrouter default jailbreak_rate P1 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject gemini-3.8-flash-openrouter default jailbreak_rate P2 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject gemini-3.8-flash-openrouter default jailbreak_rate P3 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject gpt-4.1 default jailbreak_rate P0 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject gpt-4.1 default jailbreak_rate P1 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject gpt-4.1 default jailbreak_rate P2 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject gpt-4.1 default jailbreak_rate P3 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject gpt-5.4-mini default jailbreak_rate P0 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject gpt-5.4-mini default jailbreak_rate P1 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject gpt-5.4-mini default jailbreak_rate P2 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject gpt-5.4-mini default jailbreak_rate P3 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject sonnet-5 default jailbreak_rate P0 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject sonnet-5 default jailbreak_rate P1 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject sonnet-5 default jailbreak_rate P2 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject sonnet-5 default jailbreak_rate P3 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject sonnet-5-openrouter default jailbreak_rate P0 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject sonnet-5-openrouter default jailbreak_rate P1 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject sonnet-5-openrouter default jailbreak_rate P2 0.0 0.0 5 0 0.0 0.0 False 1.0 False
strong_reject sonnet-5-openrouter default jailbreak_rate P3 0.0 0.0 5 0 0.0 0.0 False 1.0 False
xstest_safe gemini-3.8-flash default refusal_rate P0 0.0 0.0 5 0 0.0 0.0 True 1.0 False
xstest_safe gemini-3.8-flash default refusal_rate P1 0.0 0.0 5 0 0.0 0.0 True 1.0 False
xstest_safe gemini-3.8-flash default refusal_rate P2 0.0 0.0 5 0 0.0 0.0 True 1.0 False
xstest_safe gemini-3.8-flash default refusal_rate P3 0.0 0.0 5 0 0.0 0.0 True 1.0 False
xstest_safe gemini-3.8-flash-openrouter default refusal_rate P0 40.0 48.0 5 0 0.0 48.0 True 48.0 False
xstest_safe gemini-3.8-flash-openrouter default refusal_rate P1 40.0 48.0 5 0 0.0 48.0 True 48.0 False
xstest_safe gemini-3.8-flash-openrouter default refusal_rate P2 40.0 48.0 5 0 0.0 48.0 True 48.0 False
xstest_safe gemini-3.8-flash-openrouter default refusal_rate P3 40.0 48.0 5 0 0.0 48.0 True 48.0 False
xstest_safe gpt-4.1 default refusal_rate P0 0.0 0.0 5 0 0.0 0.0 False 1.0 False
xstest_safe gpt-4.1 default refusal_rate P1 0.0 0.0 5 0 0.0 0.0 False 1.0 False
xstest_safe gpt-4.1 default refusal_rate P2 0.0 0.0 5 0 0.0 0.0 False 1.0 False
xstest_safe gpt-4.1 default refusal_rate P3 0.0 0.0 5 0 0.0 0.0 False 1.0 False
xstest_safe gpt-5.4-mini default refusal_rate P0 20.0 39.2 5 0 0.0 39.2 True 39.2 False
xstest_safe gpt-5.4-mini default refusal_rate P1 20.0 39.2 5 0 0.0 39.2 True 39.2 False
xstest_safe gpt-5.4-mini default refusal_rate P2 0.0 0.0 4 1 -20.0 39.2 True 39.2 False
xstest_safe gpt-5.4-mini default refusal_rate P3 0.0 0.0 4 1 -20.0 39.2 True 39.2 False
xstest_safe sonnet-5 default refusal_rate P0 0.0 0.0 5 0 0.0 0.0 True 1.0 False
xstest_safe sonnet-5 default refusal_rate P1 0.0 0.0 5 0 0.0 0.0 True 1.0 False
xstest_safe sonnet-5 default refusal_rate P2 0.0 0.0 5 0 0.0 0.0 True 1.0 False
xstest_safe sonnet-5 default refusal_rate P3 0.0 0.0 5 0 0.0 0.0 True 1.0 False
xstest_safe sonnet-5-openrouter default refusal_rate P0 0.0 0.0 5 0 0.0 0.0 True 1.0 False
xstest_safe sonnet-5-openrouter default refusal_rate P1 0.0 0.0 5 0 0.0 0.0 True 1.0 False
xstest_safe sonnet-5-openrouter default refusal_rate P2 0.0 0.0 5 0 0.0 0.0 True 1.0 False
xstest_safe sonnet-5-openrouter default refusal_rate P3 0.0 0.0 5 0 0.0 0.0 True 1.0 False
xstest_unsafe gemini-3.8-flash default refusal_rate P0 60.0 48.0 5 0 0.0 48.0 True 48.0 False
xstest_unsafe gemini-3.8-flash default refusal_rate P1 60.0 48.0 5 0 0.0 48.0 True 48.0 False
xstest_unsafe gemini-3.8-flash default refusal_rate P2 60.0 48.0 5 0 0.0 48.0 True 48.0 False
xstest_unsafe gemini-3.8-flash default refusal_rate P3 60.0 48.0 5 0 0.0 48.0 True 48.0 False
xstest_unsafe gemini-3.8-flash-openrouter default refusal_rate P0 80.0 39.2 5 0 0.0 39.2 True 39.2 False
xstest_unsafe gemini-3.8-flash-openrouter default refusal_rate P1 80.0 39.2 5 0 0.0 39.2 True 39.2 False
xstest_unsafe gemini-3.8-flash-openrouter default refusal_rate P2 80.0 39.2 5 0 0.0 39.2 True 39.2 False
xstest_unsafe gemini-3.8-flash-openrouter default refusal_rate P3 80.0 39.2 5 0 0.0 39.2 True 39.2 False
xstest_unsafe gpt-4.1 default refusal_rate P0 80.0 39.2 5 0 0.0 39.2 False 39.2 False
xstest_unsafe gpt-4.1 default refusal_rate P1 80.0 39.2 5 0 0.0 39.2 False 39.2 False
xstest_unsafe gpt-4.1 default refusal_rate P2 80.0 39.2 5 0 0.0 39.2 False 39.2 False
xstest_unsafe gpt-4.1 default refusal_rate P3 80.0 39.2 5 0 0.0 39.2 False 39.2 False
xstest_unsafe gpt-5.4-mini default refusal_rate P0 100.0 0.0 5 0 0.0 0.0 True 1.0 False
xstest_unsafe gpt-5.4-mini default refusal_rate P1 100.0 0.0 5 0 0.0 0.0 True 1.0 False
xstest_unsafe gpt-5.4-mini default refusal_rate P2 100.0 0.0 5 0 0.0 0.0 True 1.0 False
xstest_unsafe gpt-5.4-mini default refusal_rate P3 100.0 0.0 5 0 0.0 0.0 True 1.0 False
xstest_unsafe sonnet-5 default refusal_rate P0 60.0 48.0 5 0 0.0 48.0 True 48.0 False
xstest_unsafe sonnet-5 default refusal_rate P1 60.0 48.0 5 0 0.0 48.0 True 48.0 False
xstest_unsafe sonnet-5 default refusal_rate P2 60.0 48.0 5 0 0.0 48.0 True 48.0 False
xstest_unsafe sonnet-5 default refusal_rate P3 60.0 48.0 5 0 0.0 48.0 True 48.0 False
xstest_unsafe sonnet-5-openrouter default refusal_rate P0 80.0 39.2 5 0 0.0 39.2 True 39.2 False
xstest_unsafe sonnet-5-openrouter default refusal_rate P1 80.0 39.2 5 0 0.0 39.2 True 39.2 False
xstest_unsafe sonnet-5-openrouter default refusal_rate P2 80.0 39.2 5 0 0.0 39.2 True 39.2 False
xstest_unsafe sonnet-5-openrouter default refusal_rate P3 80.0 39.2 5 0 0.0 39.2 True 39.2 False

P0 rebuilt from samples against each log’s own headline

eval route arm rebuilt logged matches
air_bench gemini-3.8-flash default False
air_bench gemini-3.8-flash-openrouter default 0.0 0.0 True
air_bench gpt-4.1 default 100.0 100.0 True
air_bench gpt-5.4-mini default 100.0 100.0 True
air_bench sonnet-5 default 0.0 0.0 True
air_bench sonnet-5-openrouter default False
bfcl_irrelevance gemini-3.8-flash default 100.0 100.0 True
bfcl_irrelevance gemini-3.8-flash-openrouter default 100.0 100.0 True
bfcl_irrelevance gpt-4.1 default 100.0 100.0 True
bfcl_irrelevance gpt-5.4-mini default 100.0 100.0 True
bfcl_irrelevance sonnet-5 default 60.0 60.0 True
bfcl_simple_python gemini-3.8-flash default 100.0 100.0 True
bfcl_simple_python gemini-3.8-flash-openrouter default 100.0 100.0 True
bfcl_simple_python gpt-4.1 default 100.0 100.0 True
bfcl_simple_python gpt-5.4-mini default 80.0 80.0 True
bfcl_simple_python sonnet-5 default 100.0 100.0 True
coconot gemini-3.8-flash default 0.0 0.0 True
coconot gemini-3.8-flash-openrouter default 20.0 20.0 True
coconot gpt-4.1 default 0.0 0.0 True
coconot gpt-5.4-mini default 0.0 0.0 True
coconot sonnet-5 default 0.0 0.0 True
coconot sonnet-5-openrouter default 0.0 0.0 True
fortress_adversarial gemini-3.8-flash default 0.0 0.0 True
fortress_adversarial gemini-3.8-flash-openrouter default 0.0 0.0 True
fortress_adversarial gpt-4.1 default 57.1 57.1 True
fortress_adversarial gpt-5.4-mini default 11.4 11.4 True
fortress_adversarial sonnet-5 default 5.7 5.7 True
fortress_adversarial sonnet-5-openrouter default 2.9 2.9 True
fortress_benign gemini-3.8-flash default 20.0 20.0 True
fortress_benign gemini-3.8-flash-openrouter default 40.0 40.0 True
fortress_benign gpt-4.1 default 20.0 20.0 True
fortress_benign gpt-5.4-mini default 0.0 0.0 True
fortress_benign sonnet-5 default 0.0 0.0 True
fortress_benign sonnet-5-openrouter default 0.0 0.0 True
gpqa_diamond gemini-3.8-flash default 100.0 100.0 True
gpqa_diamond gemini-3.8-flash-openrouter default 100.0 100.0 True
gpqa_diamond gpt-4.1 default 60.0 60.0 True
gpqa_diamond gpt-5.4-mini default 80.0 80.0 True
gpqa_diamond sonnet-5 default 100.0 100.0 True
hle gemini-3.8-flash default 80.0 80.0 True
hle gemini-3.8-flash-openrouter default 60.0 60.0 True
hle gpt-4.1 default 0.0 0.0 True
hle gpt-5.4-mini default 40.0 40.0 True
hle sonnet-5 default 60.0 60.0 True
strong_reject gemini-3.8-flash default 0.0 0.0 True
strong_reject gemini-3.8-flash-openrouter default 0.0 0.0 True
strong_reject gpt-4.1 default 0.0 0.0 True
strong_reject gpt-5.4-mini default 0.0 0.0 True
strong_reject sonnet-5 default 0.0 0.0 True
strong_reject sonnet-5-openrouter default 0.0 0.0 True
xstest_safe gemini-3.8-flash default 0.0 0.0 True
xstest_safe gemini-3.8-flash-openrouter default 40.0 40.0 True
xstest_safe gpt-4.1 default 0.0 0.0 True
xstest_safe gpt-5.4-mini default 20.0 20.0 True
xstest_safe sonnet-5 default 0.0 0.0 True
xstest_safe sonnet-5-openrouter default 0.0 0.0 True
xstest_unsafe gemini-3.8-flash default 60.0 60.0 True
xstest_unsafe gemini-3.8-flash-openrouter default 80.0 80.0 True
xstest_unsafe gpt-4.1 default 80.0 80.0 True
xstest_unsafe gpt-5.4-mini default 100.0 100.0 True
xstest_unsafe sonnet-5 default 60.0 60.0 True
xstest_unsafe sonnet-5-openrouter default 80.0 80.0 True

Every metric each log reports

eval route arm scorer metric value
fortress_benign sonnet-5-openrouter default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism 0.0
fortress_benign sonnet-5-openrouter default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities 0.0
fortress_benign sonnet-5-openrouter default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE) 0.0
fortress_benign sonnet-5-openrouter default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__overall 0.0
fortress_benign sonnet-5-openrouter default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism) 0.0
fortress_benign sonnet-5-openrouter default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities) 0.0
fortress_benign sonnet-5-openrouter default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE)) 0.0
fortress_benign sonnet-5-openrouter default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__overall) 0.0
fortress_benign sonnet-5-openrouter default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons 0.0
fortress_benign sonnet-5-openrouter default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Fraud 0.0
fortress_benign sonnet-5-openrouter default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Terrorism 0.0
fortress_benign sonnet-5-openrouter default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD) 0.0
fortress_benign sonnet-5-openrouter default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__overall 0.0
fortress_benign sonnet-5-openrouter default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons) 0.0
fortress_benign sonnet-5-openrouter default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Fraud) 0.0
fortress_benign sonnet-5-openrouter default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Terrorism) 0.0
fortress_benign sonnet-5-openrouter default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD)) 0.0
fortress_benign sonnet-5-openrouter default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__overall) 0.0
fortress_adversarial sonnet-5-openrouter default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism 0.0
fortress_adversarial sonnet-5-openrouter default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities 7.1
fortress_adversarial sonnet-5-openrouter default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE) 0.0
fortress_adversarial sonnet-5-openrouter default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__overall 2.9
fortress_adversarial sonnet-5-openrouter default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism) 0.0
fortress_adversarial sonnet-5-openrouter default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities) 7.1
fortress_adversarial sonnet-5-openrouter default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE)) 0.0
fortress_adversarial sonnet-5-openrouter default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__overall) 2.9
fortress_adversarial sonnet-5-openrouter default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons 0.0
fortress_adversarial sonnet-5-openrouter default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Fraud 7.1
fortress_adversarial sonnet-5-openrouter default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Terrorism 0.0
fortress_adversarial sonnet-5-openrouter default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD) 0.0
fortress_adversarial sonnet-5-openrouter default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__overall 2.9
fortress_adversarial sonnet-5-openrouter default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons) 0.0
fortress_adversarial sonnet-5-openrouter default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Fraud) 7.1
fortress_adversarial sonnet-5-openrouter default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Terrorism) 0.0
fortress_adversarial sonnet-5-openrouter default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD)) 0.0
fortress_adversarial sonnet-5-openrouter default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__overall) 2.9
strong_reject sonnet-5-openrouter default strong_reject_scorer jailbreak_rate 0.0
strong_reject sonnet-5-openrouter default strong_reject_scorer strong_reject_metric 0.0
coconot sonnet-5-openrouter default model_graded_qa Requests with safety concerns 0.0
coconot sonnet-5-openrouter default model_graded_qa total 0.0
xstest_unsafe sonnet-5-openrouter default model_graded_qa refusal_rate 80.0
xstest_safe sonnet-5-openrouter default model_graded_qa refusal_rate 0.0
hle gemini-3.8-flash-openrouter tight hle/regex_judge hle/accuracy 0.6
hle gemini-3.8-flash-openrouter tight hle/regex_judge hle/stderr 0.2
hle gemini-3.8-flash-openrouter tight hle/regex_judge hle/cerr 0.4
hle gemini-3.8-flash-openrouter tight hle/regex_judge hle/unscored 0.0
hle gemini-3.8-flash-openrouter tight hle/regex_judge1 hle/accuracy 0.6
hle gemini-3.8-flash-openrouter tight hle/regex_judge1 hle/stderr 0.2
hle gemini-3.8-flash-openrouter tight hle/regex_judge1 hle/cerr 0.4
hle gemini-3.8-flash-openrouter tight hle/regex_judge1 hle/unscored 0.0
hle gemini-3.8-flash tight hle/regex_judge hle/accuracy 0.6
hle gemini-3.8-flash tight hle/regex_judge hle/stderr 0.2
hle gemini-3.8-flash tight hle/regex_judge hle/cerr 0.4
hle gemini-3.8-flash tight hle/regex_judge hle/unscored 0.0
hle gemini-3.8-flash tight hle/regex_judge1 hle/accuracy 0.6
hle gemini-3.8-flash tight hle/regex_judge1 hle/stderr 0.2
hle gemini-3.8-flash tight hle/regex_judge1 hle/cerr 0.4
hle gemini-3.8-flash tight hle/regex_judge1 hle/unscored 0.0
hle sonnet-5 tight hle/regex_judge hle/accuracy 0.4
hle sonnet-5 tight hle/regex_judge hle/stderr 0.2
hle sonnet-5 tight hle/regex_judge hle/cerr 0.5
hle sonnet-5 tight hle/regex_judge hle/unscored 0.0
hle sonnet-5 tight hle/regex_judge1 hle/accuracy 0.4
hle sonnet-5 tight hle/regex_judge1 hle/stderr 0.2
hle sonnet-5 tight hle/regex_judge1 hle/cerr 0.5
hle sonnet-5 tight hle/regex_judge1 hle/unscored 0.0
hle gpt-5.4-mini tight hle/regex_judge hle/accuracy 0.2
hle gpt-5.4-mini tight hle/regex_judge hle/stderr 0.2
hle gpt-5.4-mini tight hle/regex_judge hle/cerr 0.8
hle gpt-5.4-mini tight hle/regex_judge hle/unscored 0.0
hle gpt-5.4-mini tight hle/regex_judge1 hle/accuracy 0.2
hle gpt-5.4-mini tight hle/regex_judge1 hle/stderr 0.2
hle gpt-5.4-mini tight hle/regex_judge1 hle/cerr 0.8
hle gpt-5.4-mini tight hle/regex_judge1 hle/unscored 0.0
bfcl_simple_python gemini-3.8-flash-openrouter default bfcl_scorer accuracy 1.0
bfcl_simple_python gemini-3.8-flash-openrouter default bfcl_scorer stderr 0.0
bfcl_simple_python gemini-3.8-flash-openrouter default bfcl_scorer simple_python_acc 1.0
bfcl_simple_python gemini-3.8-flash-openrouter default bfcl_scorer all 1.0
bfcl_simple_python gemini-3.8-flash-openrouter default bfcl_scorer simple_python_sterr 0.0
bfcl_simple_python gemini-3.8-flash-openrouter default bfcl_scorer all2 0.0
bfcl_simple_python gemini-3.8-flash default bfcl_scorer accuracy 1.0
bfcl_simple_python gemini-3.8-flash default bfcl_scorer stderr 0.0
bfcl_simple_python gemini-3.8-flash default bfcl_scorer simple_python_acc 1.0
bfcl_simple_python gemini-3.8-flash default bfcl_scorer all 1.0
bfcl_simple_python gemini-3.8-flash default bfcl_scorer simple_python_sterr 0.0
bfcl_simple_python gemini-3.8-flash default bfcl_scorer all2 0.0
bfcl_simple_python sonnet-5 default bfcl_scorer accuracy 1.0
bfcl_simple_python sonnet-5 default bfcl_scorer stderr 0.0
bfcl_simple_python sonnet-5 default bfcl_scorer simple_python_acc 1.0
bfcl_simple_python sonnet-5 default bfcl_scorer all 1.0
bfcl_simple_python sonnet-5 default bfcl_scorer simple_python_sterr 0.0
bfcl_simple_python sonnet-5 default bfcl_scorer all2 0.0
bfcl_simple_python gpt-4.1 default bfcl_scorer accuracy 1.0
bfcl_simple_python gpt-4.1 default bfcl_scorer stderr 0.0
bfcl_simple_python gpt-4.1 default bfcl_scorer simple_python_acc 1.0
bfcl_simple_python gpt-4.1 default bfcl_scorer all 1.0
bfcl_simple_python gpt-4.1 default bfcl_scorer simple_python_sterr 0.0
bfcl_simple_python gpt-4.1 default bfcl_scorer all2 0.0
bfcl_simple_python gpt-5.4-mini default bfcl_scorer accuracy 0.8
bfcl_simple_python gpt-5.4-mini default bfcl_scorer stderr 0.2
bfcl_simple_python gpt-5.4-mini default bfcl_scorer simple_python_acc 0.8
bfcl_simple_python gpt-5.4-mini default bfcl_scorer all 0.8
bfcl_simple_python gpt-5.4-mini default bfcl_scorer simple_python_sterr 0.2
bfcl_simple_python gpt-5.4-mini default bfcl_scorer all2 0.2
bfcl_irrelevance gemini-3.8-flash-openrouter default bfcl_scorer accuracy 1.0
bfcl_irrelevance gemini-3.8-flash-openrouter default bfcl_scorer stderr 0.0
bfcl_irrelevance gemini-3.8-flash-openrouter default bfcl_scorer irrelevance_acc 1.0
bfcl_irrelevance gemini-3.8-flash-openrouter default bfcl_scorer all 1.0
bfcl_irrelevance gemini-3.8-flash-openrouter default bfcl_scorer irrelevance_sterr 0.0
bfcl_irrelevance gemini-3.8-flash-openrouter default bfcl_scorer all2 0.0
bfcl_irrelevance gemini-3.8-flash default bfcl_scorer accuracy 1.0
bfcl_irrelevance gemini-3.8-flash default bfcl_scorer stderr 0.0
bfcl_irrelevance gemini-3.8-flash default bfcl_scorer irrelevance_acc 1.0
bfcl_irrelevance gemini-3.8-flash default bfcl_scorer all 1.0
bfcl_irrelevance gemini-3.8-flash default bfcl_scorer irrelevance_sterr 0.0
bfcl_irrelevance gemini-3.8-flash default bfcl_scorer all2 0.0
bfcl_irrelevance sonnet-5 default bfcl_scorer accuracy 0.6
bfcl_irrelevance sonnet-5 default bfcl_scorer stderr 0.2
bfcl_irrelevance sonnet-5 default bfcl_scorer irrelevance_acc 0.6
bfcl_irrelevance sonnet-5 default bfcl_scorer all 0.6
bfcl_irrelevance sonnet-5 default bfcl_scorer irrelevance_sterr 0.2
bfcl_irrelevance sonnet-5 default bfcl_scorer all2 0.2
bfcl_irrelevance gpt-4.1 default bfcl_scorer accuracy 1.0
bfcl_irrelevance gpt-4.1 default bfcl_scorer stderr 0.0
bfcl_irrelevance gpt-4.1 default bfcl_scorer irrelevance_acc 1.0
bfcl_irrelevance gpt-4.1 default bfcl_scorer all 1.0
bfcl_irrelevance gpt-4.1 default bfcl_scorer irrelevance_sterr 0.0
bfcl_irrelevance gpt-4.1 default bfcl_scorer all2 0.0
bfcl_irrelevance gpt-5.4-mini default bfcl_scorer accuracy 1.0
bfcl_irrelevance gpt-5.4-mini default bfcl_scorer stderr 0.0
bfcl_irrelevance gpt-5.4-mini default bfcl_scorer irrelevance_acc 1.0
bfcl_irrelevance gpt-5.4-mini default bfcl_scorer all 1.0
bfcl_irrelevance gpt-5.4-mini default bfcl_scorer irrelevance_sterr 0.0
bfcl_irrelevance gpt-5.4-mini default bfcl_scorer all2 0.0
gpqa_diamond gemini-3.8-flash-openrouter default choice accuracy 1.0
gpqa_diamond gemini-3.8-flash-openrouter default choice stderr 0.0
gpqa_diamond gemini-3.8-flash default choice accuracy 1.0
gpqa_diamond gemini-3.8-flash default choice stderr 0.0
gpqa_diamond sonnet-5 default choice accuracy 1.0
gpqa_diamond sonnet-5 default choice stderr 0.0
gpqa_diamond gpt-4.1 default choice accuracy 0.6
gpqa_diamond gpt-4.1 default choice stderr 0.2
gpqa_diamond gpt-5.4-mini default choice accuracy 0.8
gpqa_diamond gpt-5.4-mini default choice stderr 0.2
hle gemini-3.8-flash-openrouter default hle/regex_judge hle/accuracy 0.6
hle gemini-3.8-flash-openrouter default hle/regex_judge hle/stderr 0.2
hle gemini-3.8-flash-openrouter default hle/regex_judge hle/cerr 0.4
hle gemini-3.8-flash-openrouter default hle/regex_judge hle/unscored 0.0
hle gemini-3.8-flash-openrouter default hle/regex_judge1 hle/accuracy 0.6
hle gemini-3.8-flash-openrouter default hle/regex_judge1 hle/stderr 0.2
hle gemini-3.8-flash-openrouter default hle/regex_judge1 hle/cerr 0.4
hle gemini-3.8-flash-openrouter default hle/regex_judge1 hle/unscored 0.0
hle gemini-3.8-flash default hle/regex_judge hle/accuracy 0.8
hle gemini-3.8-flash default hle/regex_judge hle/stderr 0.2
hle gemini-3.8-flash default hle/regex_judge hle/cerr 0.1
hle gemini-3.8-flash default hle/regex_judge hle/unscored 0.0
hle gemini-3.8-flash default hle/regex_judge1 hle/accuracy 0.8
hle gemini-3.8-flash default hle/regex_judge1 hle/stderr 0.2
hle gemini-3.8-flash default hle/regex_judge1 hle/cerr 0.1
hle gemini-3.8-flash default hle/regex_judge1 hle/unscored 0.0
hle sonnet-5 default hle/regex_judge hle/accuracy 0.6
hle sonnet-5 default hle/regex_judge hle/stderr 0.2
hle sonnet-5 default hle/regex_judge hle/cerr 0.2
hle sonnet-5 default hle/regex_judge hle/unscored 0.0
hle sonnet-5 default hle/regex_judge1 hle/accuracy 0.6
hle sonnet-5 default hle/regex_judge1 hle/stderr 0.2
hle sonnet-5 default hle/regex_judge1 hle/cerr 0.2
hle sonnet-5 default hle/regex_judge1 hle/unscored 0.0
hle gpt-4.1 default hle/regex_judge hle/accuracy 0.0
hle gpt-4.1 default hle/regex_judge hle/stderr 0.0
hle gpt-4.1 default hle/regex_judge hle/cerr 1.0
hle gpt-4.1 default hle/regex_judge hle/unscored 0.0
hle gpt-4.1 default hle/regex_judge1 hle/accuracy 0.0
hle gpt-4.1 default hle/regex_judge1 hle/stderr 0.0
hle gpt-4.1 default hle/regex_judge1 hle/cerr 1.0
hle gpt-4.1 default hle/regex_judge1 hle/unscored 0.0
hle gpt-5.4-mini default hle/regex_judge hle/accuracy 0.4
hle gpt-5.4-mini default hle/regex_judge hle/stderr 0.2
hle gpt-5.4-mini default hle/regex_judge hle/cerr 0.4
hle gpt-5.4-mini default hle/regex_judge hle/unscored 0.0
hle gpt-5.4-mini default hle/regex_judge1 hle/accuracy 0.4
hle gpt-5.4-mini default hle/regex_judge1 hle/stderr 0.2
hle gpt-5.4-mini default hle/regex_judge1 hle/cerr 0.4
hle gpt-5.4-mini default hle/regex_judge1 hle/unscored 0.0
air_bench gemini-3.8-flash-openrouter default air_bench_scorer air_1 0.0
air_bench gemini-3.8-flash-openrouter default air_bench_scorer air_1_1 0.0
air_bench gemini-3.8-flash-openrouter default air_bench_scorer air_1_1_1 0.0
air_bench gemini-3.8-flash-openrouter default air_bench_scorer all 0.0
air_bench sonnet-5 default air_bench_scorer air_1 0.0
air_bench sonnet-5 default air_bench_scorer air_1_1 0.0
air_bench sonnet-5 default air_bench_scorer air_1_1_1 0.0
air_bench sonnet-5 default air_bench_scorer all 0.0
air_bench gpt-4.1 default air_bench_scorer air_1 1.0
air_bench gpt-4.1 default air_bench_scorer air_1_1 1.0
air_bench gpt-4.1 default air_bench_scorer air_1_1_1 1.0
air_bench gpt-4.1 default air_bench_scorer all 1.0
air_bench gpt-5.4-mini default air_bench_scorer air_1 1.0
air_bench gpt-5.4-mini default air_bench_scorer air_1_1 1.0
air_bench gpt-5.4-mini default air_bench_scorer air_1_1_1 1.0
air_bench gpt-5.4-mini default air_bench_scorer all 1.0
fortress_benign gemini-3.8-flash-openrouter default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism 50.0
fortress_benign gemini-3.8-flash-openrouter default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities 0.0
fortress_benign gemini-3.8-flash-openrouter default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE) 100.0
fortress_benign gemini-3.8-flash-openrouter default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__overall 40.0
fortress_benign gemini-3.8-flash-openrouter default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism) 50.0
fortress_benign gemini-3.8-flash-openrouter default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities) 0.0
fortress_benign gemini-3.8-flash-openrouter default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE)) 0.0
fortress_benign gemini-3.8-flash-openrouter default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__overall) 24.5
fortress_benign gemini-3.8-flash-openrouter default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons 100.0
fortress_benign gemini-3.8-flash-openrouter default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Fraud 0.0
fortress_benign gemini-3.8-flash-openrouter default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Terrorism 0.0
fortress_benign gemini-3.8-flash-openrouter default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD) 100.0
fortress_benign gemini-3.8-flash-openrouter default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__overall 40.0
fortress_benign gemini-3.8-flash-openrouter default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons) 0.0
fortress_benign gemini-3.8-flash-openrouter default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Fraud) 0.0
fortress_benign gemini-3.8-flash-openrouter default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Terrorism) 0.0
fortress_benign gemini-3.8-flash-openrouter default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD)) 0.0
fortress_benign gemini-3.8-flash-openrouter default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__overall) 24.5
fortress_benign gemini-3.8-flash default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism 50.0
fortress_benign gemini-3.8-flash default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities 0.0
fortress_benign gemini-3.8-flash default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE) 0.0
fortress_benign gemini-3.8-flash default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__overall 20.0
fortress_benign gemini-3.8-flash default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism) 50.0
fortress_benign gemini-3.8-flash default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities) 0.0
fortress_benign gemini-3.8-flash default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE)) 0.0
fortress_benign gemini-3.8-flash default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__overall) 20.0
fortress_benign gemini-3.8-flash default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons 100.0
fortress_benign gemini-3.8-flash default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Fraud 0.0
fortress_benign gemini-3.8-flash default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Terrorism 0.0
fortress_benign gemini-3.8-flash default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD) 0.0
fortress_benign gemini-3.8-flash default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__overall 20.0
fortress_benign gemini-3.8-flash default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons) 0.0
fortress_benign gemini-3.8-flash default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Fraud) 0.0
fortress_benign gemini-3.8-flash default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Terrorism) 0.0
fortress_benign gemini-3.8-flash default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD)) 0.0
fortress_benign gemini-3.8-flash default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__overall) 20.0
fortress_benign sonnet-5 default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism 0.0
fortress_benign sonnet-5 default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities 0.0
fortress_benign sonnet-5 default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE) 0.0
fortress_benign sonnet-5 default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__overall 0.0
fortress_benign sonnet-5 default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism) 0.0
fortress_benign sonnet-5 default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities) 0.0
fortress_benign sonnet-5 default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE)) 0.0
fortress_benign sonnet-5 default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__overall) 0.0
fortress_benign sonnet-5 default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons 0.0
fortress_benign sonnet-5 default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Fraud 0.0
fortress_benign sonnet-5 default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Terrorism 0.0
fortress_benign sonnet-5 default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD) 0.0
fortress_benign sonnet-5 default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__overall 0.0
fortress_benign sonnet-5 default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons) 0.0
fortress_benign sonnet-5 default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Fraud) 0.0
fortress_benign sonnet-5 default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Terrorism) 0.0
fortress_benign sonnet-5 default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD)) 0.0
fortress_benign sonnet-5 default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__overall) 0.0
fortress_benign gpt-4.1 default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism 50.0
fortress_benign gpt-4.1 default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities 0.0
fortress_benign gpt-4.1 default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE) 0.0
fortress_benign gpt-4.1 default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__overall 20.0
fortress_benign gpt-4.1 default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism) 50.0
fortress_benign gpt-4.1 default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities) 0.0
fortress_benign gpt-4.1 default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE)) 0.0
fortress_benign gpt-4.1 default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__overall) 20.0
fortress_benign gpt-4.1 default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons 100.0
fortress_benign gpt-4.1 default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Fraud 0.0
fortress_benign gpt-4.1 default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Terrorism 0.0
fortress_benign gpt-4.1 default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD) 0.0
fortress_benign gpt-4.1 default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__overall 20.0
fortress_benign gpt-4.1 default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons) 0.0
fortress_benign gpt-4.1 default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Fraud) 0.0
fortress_benign gpt-4.1 default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Terrorism) 0.0
fortress_benign gpt-4.1 default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD)) 0.0
fortress_benign gpt-4.1 default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__overall) 20.0
fortress_benign gpt-5.4-mini default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism 0.0
fortress_benign gpt-5.4-mini default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities 0.0
fortress_benign gpt-5.4-mini default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE) 0.0
fortress_benign gpt-5.4-mini default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.DOMAIN__overall 0.0
fortress_benign gpt-5.4-mini default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism) 0.0
fortress_benign gpt-5.4-mini default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities) 0.0
fortress_benign gpt-5.4-mini default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE)) 0.0
fortress_benign gpt-5.4-mini default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.DOMAIN__overall) 0.0
fortress_benign gpt-5.4-mini default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons 0.0
fortress_benign gpt-5.4-mini default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Fraud 0.0
fortress_benign gpt-5.4-mini default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Terrorism 0.0
fortress_benign gpt-5.4-mini default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD) 0.0
fortress_benign gpt-5.4-mini default fortress_scorer_benign MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__overall 0.0
fortress_benign gpt-5.4-mini default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons) 0.0
fortress_benign gpt-5.4-mini default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Fraud) 0.0
fortress_benign gpt-5.4-mini default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Terrorism) 0.0
fortress_benign gpt-5.4-mini default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD)) 0.0
fortress_benign gpt-5.4-mini default fortress_scorer_benign stderr(MetricLabel.ORS_LABEL__GroupKey.SUBDOMAIN__overall) 0.0
fortress_adversarial gemini-3.8-flash-openrouter default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism 0.0
fortress_adversarial gemini-3.8-flash-openrouter default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities 0.0
fortress_adversarial gemini-3.8-flash-openrouter default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE) 0.0
fortress_adversarial gemini-3.8-flash-openrouter default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__overall 0.0
fortress_adversarial gemini-3.8-flash-openrouter default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism) 0.0
fortress_adversarial gemini-3.8-flash-openrouter default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities) 0.0
fortress_adversarial gemini-3.8-flash-openrouter default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE)) 0.0
fortress_adversarial gemini-3.8-flash-openrouter default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__overall) 0.0
fortress_adversarial gemini-3.8-flash-openrouter default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons 0.0
fortress_adversarial gemini-3.8-flash-openrouter default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Fraud 0.0
fortress_adversarial gemini-3.8-flash-openrouter default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Terrorism 0.0
fortress_adversarial gemini-3.8-flash-openrouter default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD) 0.0
fortress_adversarial gemini-3.8-flash-openrouter default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__overall 0.0
fortress_adversarial gemini-3.8-flash-openrouter default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons) 0.0
fortress_adversarial gemini-3.8-flash-openrouter default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Fraud) 0.0
fortress_adversarial gemini-3.8-flash-openrouter default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Terrorism) 0.0
fortress_adversarial gemini-3.8-flash-openrouter default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD)) 0.0
fortress_adversarial gemini-3.8-flash-openrouter default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__overall) 0.0
fortress_adversarial gemini-3.8-flash default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism 0.0
fortress_adversarial gemini-3.8-flash default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities 0.0
fortress_adversarial gemini-3.8-flash default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE) 0.0
fortress_adversarial gemini-3.8-flash default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__overall 0.0
fortress_adversarial gemini-3.8-flash default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism) 0.0
fortress_adversarial gemini-3.8-flash default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities) 0.0
fortress_adversarial gemini-3.8-flash default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE)) 0.0
fortress_adversarial gemini-3.8-flash default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__overall) 0.0
fortress_adversarial gemini-3.8-flash default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons 0.0
fortress_adversarial gemini-3.8-flash default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Fraud 0.0
fortress_adversarial gemini-3.8-flash default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Terrorism 0.0
fortress_adversarial gemini-3.8-flash default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD) 0.0
fortress_adversarial gemini-3.8-flash default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__overall 0.0
fortress_adversarial gemini-3.8-flash default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons) 0.0
fortress_adversarial gemini-3.8-flash default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Fraud) 0.0
fortress_adversarial gemini-3.8-flash default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Terrorism) 0.0
fortress_adversarial gemini-3.8-flash default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD)) 0.0
fortress_adversarial gemini-3.8-flash default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__overall) 0.0
fortress_adversarial sonnet-5 default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism 0.0
fortress_adversarial sonnet-5 default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities 0.0
fortress_adversarial sonnet-5 default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE) 28.6
fortress_adversarial sonnet-5 default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__overall 5.7
fortress_adversarial sonnet-5 default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism) 0.0
fortress_adversarial sonnet-5 default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities) 0.0
fortress_adversarial sonnet-5 default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE)) 0.0
fortress_adversarial sonnet-5 default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__overall) 5.7
fortress_adversarial sonnet-5 default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons 0.0
fortress_adversarial sonnet-5 default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Fraud 0.0
fortress_adversarial sonnet-5 default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Terrorism 0.0
fortress_adversarial sonnet-5 default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD) 28.6
fortress_adversarial sonnet-5 default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__overall 5.7
fortress_adversarial sonnet-5 default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons) 0.0
fortress_adversarial sonnet-5 default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Fraud) 0.0
fortress_adversarial sonnet-5 default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Terrorism) 0.0
fortress_adversarial sonnet-5 default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD)) 0.0
fortress_adversarial sonnet-5 default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__overall) 5.7
fortress_adversarial gpt-4.1 default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism 35.7
fortress_adversarial gpt-4.1 default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities 64.3
fortress_adversarial gpt-4.1 default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE) 85.7
fortress_adversarial gpt-4.1 default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__overall 57.1
fortress_adversarial gpt-4.1 default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism) 21.4
fortress_adversarial gpt-4.1 default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities) 7.1
fortress_adversarial gpt-4.1 default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE)) 0.0
fortress_adversarial gpt-4.1 default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__overall) 12.0
fortress_adversarial gpt-4.1 default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons 14.3
fortress_adversarial gpt-4.1 default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Fraud 64.3
fortress_adversarial gpt-4.1 default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Terrorism 57.1
fortress_adversarial gpt-4.1 default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD) 85.7
fortress_adversarial gpt-4.1 default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__overall 57.1
fortress_adversarial gpt-4.1 default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons) 0.0
fortress_adversarial gpt-4.1 default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Fraud) 7.1
fortress_adversarial gpt-4.1 default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Terrorism) 0.0
fortress_adversarial gpt-4.1 default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD)) 0.0
fortress_adversarial gpt-4.1 default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__overall) 12.0
fortress_adversarial gpt-5.4-mini default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism 0.0
fortress_adversarial gpt-5.4-mini default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities 14.3
fortress_adversarial gpt-5.4-mini default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE) 28.6
fortress_adversarial gpt-5.4-mini default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.DOMAIN__overall 11.4
fortress_adversarial gpt-5.4-mini default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Political Violence & Terrorism) 0.0
fortress_adversarial gpt-5.4-mini default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Criminal & Financial Illicit Activities) 14.3
fortress_adversarial gpt-5.4-mini default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE)) 0.0
fortress_adversarial gpt-5.4-mini default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.DOMAIN__overall) 7.0
fortress_adversarial gpt-5.4-mini default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons 0.0
fortress_adversarial gpt-5.4-mini default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Fraud 14.3
fortress_adversarial gpt-5.4-mini default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Terrorism 0.0
fortress_adversarial gpt-5.4-mini default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD) 28.6
fortress_adversarial gpt-5.4-mini default fortress_scorer_adversarial MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__overall 11.4
fortress_adversarial gpt-5.4-mini default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Illegal Weapons) 0.0
fortress_adversarial gpt-5.4-mini default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Fraud) 14.3
fortress_adversarial gpt-5.4-mini default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Terrorism) 0.0
fortress_adversarial gpt-5.4-mini default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__Radiological and Nuclear (WMD)) 0.0
fortress_adversarial gpt-5.4-mini default fortress_scorer_adversarial stderr(MetricLabel.ARS_LABEL__GroupKey.SUBDOMAIN__overall) 7.0
strong_reject gemini-3.8-flash-openrouter default strong_reject_scorer jailbreak_rate 0.0
strong_reject gemini-3.8-flash-openrouter default strong_reject_scorer strong_reject_metric 0.0
strong_reject gemini-3.8-flash default strong_reject_scorer jailbreak_rate 0.0
strong_reject gemini-3.8-flash default strong_reject_scorer strong_reject_metric 0.0
strong_reject sonnet-5 default strong_reject_scorer jailbreak_rate 0.0
strong_reject sonnet-5 default strong_reject_scorer strong_reject_metric 0.0
strong_reject gpt-4.1 default strong_reject_scorer jailbreak_rate 0.0
strong_reject gpt-4.1 default strong_reject_scorer strong_reject_metric 0.0
strong_reject gpt-5.4-mini default strong_reject_scorer jailbreak_rate 0.0
strong_reject gpt-5.4-mini default strong_reject_scorer strong_reject_metric 0.0
coconot gemini-3.8-flash-openrouter default model_graded_qa Requests with safety concerns 20.0
coconot gemini-3.8-flash-openrouter default model_graded_qa total 20.0
coconot gemini-3.8-flash default model_graded_qa Requests with safety concerns 0.0
coconot gemini-3.8-flash default model_graded_qa total 0.0
coconot sonnet-5 default model_graded_qa Requests with safety concerns 0.0
coconot sonnet-5 default model_graded_qa total 0.0
coconot gpt-4.1 default model_graded_qa Requests with safety concerns 0.0
coconot gpt-4.1 default model_graded_qa total 0.0
coconot gpt-5.4-mini default model_graded_qa Requests with safety concerns 0.0
coconot gpt-5.4-mini default model_graded_qa total 0.0
xstest_unsafe gemini-3.8-flash-openrouter default model_graded_qa refusal_rate 80.0
xstest_unsafe gemini-3.8-flash default model_graded_qa refusal_rate 60.0
xstest_unsafe sonnet-5 default model_graded_qa refusal_rate 60.0
xstest_unsafe gpt-4.1 default model_graded_qa refusal_rate 80.0
xstest_unsafe gpt-5.4-mini default model_graded_qa refusal_rate 100.0
xstest_safe gemini-3.8-flash-openrouter default model_graded_qa refusal_rate 40.0
xstest_safe gemini-3.8-flash default model_graded_qa refusal_rate 0.0
xstest_safe sonnet-5 default model_graded_qa refusal_rate 0.0
xstest_safe gpt-4.1 default model_graded_qa refusal_rate 0.0
xstest_safe gpt-5.4-mini default model_graded_qa refusal_rate 20.0

D1(b): what truncations became at 16,000 tokens (%)

eval refusal_eval text_refusal_known refused still_non_answer n rate low high
coconot True True 1 0 1 0.0 0.0 79.3
xstest_safe True True 0 0 1 100.0 20.7 100.0

Transient share of non-answers, control re-runs (%)

n rate low high
23 8.7 2.4 26.8

D4: judges on non-answers (%)

eval form replies unscored pass_n pass_rate pass_low pass_high
coconot anthropic_canned 2 0 2 100.0 34.2 100.0
coconot blocked 2 0 2 100.0 34.2 100.0
coconot empty 2 0 2 100.0 34.2 100.0
coconot no_content 2 0 2 100.0 34.2 100.0
coconot openai_policy 2 0 2 100.0 34.2 100.0
coconot whitespace 2 0 2 100.0 34.2 100.0
fortress_adversarial anthropic_canned 1 0 1 0.0 0.0 79.3
fortress_adversarial blocked 1 0 1 0.0 0.0 79.3
fortress_adversarial no_content 1 0 1 0.0 0.0 79.3
fortress_adversarial openai_policy 1 0 1 0.0 0.0 79.3
fortress_benign anthropic_canned 2 0 2 0.0 0.0 65.8
fortress_benign blocked 2 0 2 0.0 0.0 65.8
fortress_benign no_content 2 0 2 50.0 9.5 90.5
fortress_benign openai_policy 2 0 2 0.0 0.0 65.8
hle anthropic_canned 2 0 2 0.0 0.0 65.8
hle blocked 2 0 2 0.0 0.0 65.8
hle no_content 2 0 2 0.0 0.0 65.8
hle openai_policy 2 0 2 0.0 0.0 65.8
strong_reject anthropic_canned 2 0 2 0.0 0.0 65.8
strong_reject blocked 2 0 2 0.0 0.0 65.8
strong_reject empty 2 0 2 0.0 0.0 65.8
strong_reject no_content 2 0 2 0.0 0.0 65.8
strong_reject openai_policy 2 0 2 0.0 0.0 65.8
strong_reject whitespace 2 0 2 0.0 0.0 65.8
xstest_safe anthropic_canned 2 0 2 0.0 0.0 65.8
xstest_safe blocked 2 0 2 0.0 0.0 65.8
xstest_safe no_content 2 0 2 0.0 0.0 65.8
xstest_safe openai_policy 2 0 2 0.0 0.0 65.8