HumanWill · Cybersecurity benchmark

When security AI withholds help

False-refusal rate (%) · Lower is better · 21 model configurations

Hover, focus or tap a model for five-topic results. Scroll sideways on smaller screens.

No models selected. Choose models above or select All to restore the full chart.

020406080100

0.5%Mistral Small 26030.7%Mistral Medium 3.52.1%GPT-5.6 Sol3.5%GLM 5.3 Flash3.8%GPT-5.6 Terra6.6%Sonnet 4.66.8%GPT-6 Sol9.0%DeepSeek V4.1 Flash11.6%Meta Muse Wonder14.4%Meta Muse Spark 1.3 *18.4%Mistral Large 4.024.3%Qwen3.8 Max 090226.9%Nemotron 3 Ultra43.1%Gemini 3.8 Flash *62.7%Grok 4.764.4%Grok 4.673.1%GPT-6 Astra83.7%Claude Sonnet 5.588.0%Claude Opus 591.0%Fable 5.193.9%Claude Opus 5.5

424 selected cybersecurity scenarios per configuration. Historical comparison: model settings, provider routes and run dates vary. Not everyday refusal rates or an overall safety score. Partial/full withholding counted; missing outcomes excluded. Muse Wonder added 13 Oct 2026; its 24 persistent zero-output Meta completions are explicitly counted as false refusals, alongside 25 semantically judged text refusals. Usefulness remains scored on 400 answers. Mistral Large 4.0 added 7 Oct 2026 via OpenRouter/Mistral, with default reasoning and a 65,536-token output cap. HumanWill · CC BY 4.0.