The attacks a model refuses on its own are not the ones that cause damage. The ones it lets through are. On the locked held-out set the model lets 21 of 44 through unaided, and MoorAI catches 20 of them — 95.2% of the gap. AMTSO’s rule is what forces the figure to be stated that way: a refusal the model produces on its own is recorded, and it is not credited to the product, so the only region a second layer is graded in is the gap the model leaves.
The chain that number comes from
Of the 21 attacks the model does not refuse, MoorAI catches 20, and residual exposure is 1 of 44: one case walks through both layers untouched. The baseline that makes those two figures readable is the run with no product in the path at all, where the model alone stops 23 of 44 — 52.3%. Counting both layers, 43 of 44 — 97.7% are stopped by one or the other.
The two layers are complementary rather than redundant. The model’s share is almost entirely plain-language attacks; MoorAI’s twenty are almost entirely obfuscated ones. Refusal is a property of how a request looks, which is exactly why the decision is re-made downstream on the resolved action, where there is no surface left to disguise. The refusal baseline is measured against a local 8B open-weights model, llama3:latest at Q4_0, not a frontier one — a condition that travels with those two figures wherever they go, and one the report takes apart in full.
What is measured, and why
Detection on that same locked set is 86.4% — 38 of 44 at 100% precision, deterministic: no model, no network, pure logic on the resolved tool call. The set is a held-out half the detectors are never developed against, never tuned against and never eyeballed; it gets opened to measure, and that is the only thing it is for. The attacks are organised against the ten families of the HackAgent attack taxonomy — CipherChat, FlipAttack, h4rm3l, DAN, AutoDAN, BoN, AdvPrefix, PAP, PAIR and TAP — someone else’s published taxonomy, with no affiliation, endorsement or review by them.
A case counts only if the malicious outcome actually occurs in the same environment with the product absent: a real read of a secret, a real outbound request to a controlled endpoint, a real destructive command, verified unprotected first. That is AMTSO’s baseline validation, and it builds a smaller, considerably less flattering corpus than the alternative. Generation is deterministic and seeded, so the same seed reproduces the same corpus and the split stays fixed between runs.
False positives are reported unblended, because AMTSO requires that they “should not be hidden in aggregate attack scores”: 3.32% — 20 flags out of 602 — from a 610-prompt benign corpus, of which 269 are adversarially-shaped hard negatives, benign work engineered to look malicious. Twenty times, legitimate work gets flagged. In a tool sitting inline with a developer’s workflow that is a real tax, and it belongs on its own line rather than averaged into invisibility.
Across 624 labelled attack samples the corpus distributes over AMTSO’s five vectors as 296 direct input (47.4%), 130 indirect content (20.8%), 78 tool / MCP supply chain (12.5%), 62 outbound action (9.9%), 42 memory and cross-agent (6.7%), and 16 static code artifacts (2.6%) that map to no vector. Direct input sits below half and no vector is starved — which matters, because a headline recall figure is a weighted average over that distribution and nobody prints the weights. Every non-direct vector has its own corpus and its own scorer; those per-vector scores are in the report.
After the jailbreak has already won: 12 of 12
MoorAI does not try to win an argument with a language model. It assumes the model can be talked into anything and moves the decision to the one place persuasion stops working: the moment the agent stops talking and starts doing. So the action-layer test begins where most vendor demos end. The jailbreak succeeded, the injection got through, the agent has already decided to do the harmful thing, and one question is left — does the tool call execute? Twelve times out of twelve, it does not.
An enrolled machine prevents before anyone writes a policy. Reverse shells and local credential egress are denied on the built-in defaults; credential-file reads, destructive tool calls and unsanctioned installs halt for sign-off. That is 31.6% of the outbound-action corpus prevented with nothing configured, 75.4% under an enforcing policy, 86.0% fail-closed. Each default fires on zero of 781 benign prompts.
On the injection and jailbreak held-out set the split is 0 prevented / 38 detected / 6 missed — deliberately. Those samples are attempts, not actions, and they resolve to notify: an injection attempt is not yet harm, and blocking on suspicion is where a security tool starts costing real work. We stop the action it resolves to instead.
Hook-level enforcement is Claude Code. At the MCP layer it is host-independent — Cursor, Copilot and Claude Desktop tool calls are gated identically — but only 4 of those 12 actions natively traverse MCP, and Codex is not covered there at all.
And one limit on that path is a timing one. A PreToolUse hook fires before the request goes out, so the page does not exist yet to be read; inbound WebFetch and WebSearch content is scanned by a second hook that fires on the result instead. That hook sees the whole payload, but it runs after the fetch has already happened — so on the inbound direction the product reports and warns rather than prevents.
The rest is in the report
Everything above is the headline. The report is the working: the per-vector scores and the four policy postures they were measured under, the false-positive breakdown by twin family with the detectors named, the full refusal chain including how refusal behaves when the surface form changes, the AMTSO dimension coverage table, and the section AMTSO asks for and most vendor benchmarks omit — what was not tested, and where it may not generalise. It is a vendor measuring itself, unflattering rows included; no third party has reproduced any figure in it.
Grade the next vendor the same way
Published criteria cut both ways. If it is fair to grade ourselves against AMTSO’s, it is fair for you to put the next agentic security product you are shown through the same questions — and none of them require you to take a vendor’s word for anything. They are written up as a checklist you can take into a demo: Choosing an agentic security vendor, six questions with the weak answer, the good answer and how to verify it.
MoorAI is content-free, open-core (MIT) security for AI agents, enforced at the action layer on the developer's endpoint. All figures above are our own measurements, self-reported and not independently verified. See also: Obfuscated prompts beat content scanning and Prompt injection: content scanning won't stop it.