The Claude Sonnet figure was scored on a different path from the 86.4% headline: its script joins a multi-turn attack into one text, and so credited MoorAI with a catch that the headline’s turn-by-turn scorer misses. On the headline’s own path MoorAI catches 6 of the 8 attacks Sonnet lets through (75.0%, not 87.5%; a marginal 13.6%, not 15.9%), and 2 of 44 attacks get past both layers, not 1; the Llama 3 8B figures are unchanged.
The attacks that cause damage are the ones a model lets through. On the locked held-out set a local Llama 3 8B lets 21 of 44 through unaided and MoorAI catches 20: 95.2% of the gap, a marginal 45.5% of the corpus. The deployed Claude Sonnet assistant refuses more on its own and lets 8 through; MoorAI catches 6: 75.0% of the gap, a marginal 13.6%. AMTSO’s rule is what forces the figure to be stated that way: a refusal the model produces on its own is recorded, and it is not credited to the product, so the only region a second layer is graded in is the gap the model leaves.
The chain that number comes from
Of the 21 attacks the model does not refuse, MoorAI catches 20, and residual exposure is 1 of 44: one case walks through both layers untouched. The baseline that makes those two figures readable is the run with no product in the path at all, where the model alone stops 23 of 44 — 52.3%. Counting both layers, 43 of 44 — 97.7% are stopped by one or the other.
The two layers are complementary rather than redundant. The model’s share is almost entirely plain-language attacks; MoorAI’s twenty are almost entirely obfuscated ones. Refusal is a property of how a request looks, which is why the decision is re-made downstream on the resolved action, where there is no surface left to disguise. Those figures are measured against a local 8B open-weights model, llama3:latest at Q4_0. Against the deployed Claude Sonnet assistant, which refuses 25 of the 33 attacks it receives and whose provider rejects another 11 of 44 before the model sees them, MoorAI catches 6 of the 8 left: 75.0% of the gap and a marginal 13.6% of the corpus, against 45.5% for the 8B. A frontier assistant refuses more on its own, so less is left for MoorAI to add. The residual is 1 of 44 against the 8B and 2 of 44 against Sonnet, and the report takes both apart in full.
What is measured, and why
Detection on that same locked set is 86.4% — 38 of 44 at 100% precision, deterministic: no model, no network, pure logic on the resolved tool call. The set is a held-out half the detectors are never developed against, never tuned against and never eyeballed; it gets opened to measure, and that is the only thing it is for. The attacks are organised against the ten families of the HackAgent attack taxonomy (CipherChat, FlipAttack, h4rm3l, DAN, AutoDAN, BoN, AdvPrefix, PAP, PAIR and TAP), someone else’s published taxonomy, with no affiliation, endorsement or review by them.
A case counts only if the malicious outcome actually occurs in the same environment with the product absent: a real read of a secret, a real outbound request to a controlled endpoint, a real destructive command, verified unprotected first. That is AMTSO’s baseline validation, and it builds a smaller, considerably less flattering corpus than the alternative. Generation is deterministic and seeded, so the same seed reproduces the same corpus and the split stays fixed between runs.
False positives are reported unblended, because AMTSO requires that they “should not be hidden in aggregate attack scores”: 3.32% (20 flags out of 602) from a 610-prompt benign corpus, of which 269 are adversarially-shaped hard negatives, benign work engineered to look malicious. Twenty times, legitimate work gets flagged. In a tool sitting inline with a developer’s workflow that is a real tax, and it belongs on its own line rather than averaged into invisibility.
Across 624 labelled attack samples the corpus distributes over AMTSO’s five vectors as 296 direct input (47.4%), 130 indirect content (20.8%), 78 tool / MCP supply chain (12.5%), 62 outbound action (9.9%), 42 memory and cross-agent (6.7%), and 16 static code artifacts (2.6%) that map to no vector. Direct input sits below half and no vector is starved. That matters, because a headline recall figure is a weighted average over that distribution and nobody prints the weights. Every non-direct vector has its own corpus and its own scorer; those per-vector scores are in the report.
After the jailbreak has already won: 12 of 12
MoorAI does not try to win an argument with a language model. It assumes the model can be talked into anything and moves the decision to the one place persuasion stops working: the moment the agent stops talking and starts doing. So the action-layer test begins where most vendor demos end. The jailbreak succeeded, the injection got through, the agent has already decided to do the harmful thing, and one question is left: whether the tool call executes. In twelve cases out of twelve, it does not.
An enrolled machine prevents before anyone writes a policy. Reverse shells and local credential egress are denied on the built-in defaults; credential-file reads, destructive tool calls and unsanctioned installs halt for sign-off. That is 31.6% of the outbound-action corpus prevented with nothing configured, 75.4% under an enforcing policy, 86.0% fail-closed. Each default fires on zero of 781 benign prompts.
On the injection and jailbreak held-out set the split is 0 prevented / 38 detected / 6 missed, by design. Those samples are attempts, not actions, and they resolve to notify: an injection attempt is not yet harm, and blocking on suspicion is where a security tool starts costing real work. We stop the action it resolves to instead.
Hook-level enforcement is Claude Code. At the MCP layer it is host-independent (Cursor, Copilot and Claude Desktop tool calls are gated identically), but only 4 of those 12 actions natively traverse MCP, and Codex is not covered there at all.
One limit on that path is timing. A PreToolUse hook fires before the request goes out, so the page does not exist yet to be read; inbound WebFetch and WebSearch content is scanned by a second hook that fires on the result instead. That hook sees the whole payload, but it runs after the fetch has already happened, so on the inbound direction the product reports and warns rather than prevents.
The rest is in the report
The report has the working: the per-vector scores and the four policy postures they were measured under, the false-positive breakdown by twin family with the detectors named, the full refusal chain including how refusal behaves when the surface form changes, the AMTSO dimension coverage table, and the section AMTSO asks for and most vendor benchmarks omit: what was not tested, and where it may not generalise. It is a vendor measuring itself, and no third party has reproduced any figure in it.
Grade the next vendor the same way
Published criteria cut both ways. If it is fair to grade ourselves against AMTSO’s, it is fair for you to put the next agentic security product you are shown through the same questions, none of which require you to take a vendor’s word for anything. They are written up as a checklist you can take into a demo: Choosing an agentic security vendor, six questions with the weak answer, the good answer and how to verify it.
MoorAI is content-free, open-core (MIT) security for AI agents, enforced at the action layer on the developer's endpoint. All figures above are our own measurements, self-reported and not independently verified. See also: Obfuscated prompts beat content scanning and Prompt injection: content scanning won't stop it.