How we test
We grade MoorAI against AMTSO's Guidelines for Testing of Agentic Security Products v1.0, on a locked corpus we are not allowed to tune against, with the underlying model's own refusals measured and subtracted, so what is left is what the product adds. Every detection number on this site comes from a harness in the open-source agent repository, and the method that produces it is on this page in full.
What the method produces. Five headline figures for the current release, each with the condition it was measured under. The working behind each one (the outcome distribution, the per-family and per-vector breakdown, the per-device-state enforcement table) is in the full benchmark report.
| Result | What it was measured against |
|---|---|
| 95.2% of the model-refusal gap (Llama 3 8B) · 75.0% (Claude Sonnet) | Against the local Llama 3 8B baseline, 21 of 44 attacks are not refused unaided and MoorAI catches 20. Against the deployed Claude Sonnet assistant, only 8 of 44 get past its own refusals and the provider's platform block, and MoorAI catches 6. Corpus-wide, the marginal value is 45.5% (20/44) against the 8B and 13.6% (6/44) against Sonnet: a frontier assistant refuses more on its own, so less is left for MoorAI to add. Past both layers: 1 of 44 (2.3%) against the 8B and 2 of 44 (4.5%) against Sonnet. This is the number the refusal baseline exists to produce. Breakdown ↗ |
| 86.4% on the locked held-out set | 38 of 44 attacks on a corpus detector work is never allowed to open, at 100% precision — 0 false positives across its 10 benign samples. Section 05 is why that set is worth more than a larger score from the tune half. Breakdown ↗ |
| 3.32% false positives on benign traffic | 20 of 602, specificity 96.68% — measured on the 610-prompt benign corpus, 269 of it adversarially-shaped hard negatives written to look like the attacks they sit beside. Breakdown ↗ |
| 12 of 12 malicious tool calls stopped | Prevention, not detection: with the agent assumed already hijacked and an enforcing policy in place, every one of the twelve malicious actions it then attempted was denied by the real hook. Scoped to one agent host. Breakdown ↗ |
| 101/102 deterministic red-team coverage | Across the ten HackAgent families, with no model anywhere in the loop — byte-identical on any machine. The redteam-eval held-out run scores 100% (29/29) at 94% precision (4 false positives / 178). Breakdown ↗ |
| What this page commits us to | What it means |
|---|---|
| A locked split we never tune against | Detection logic is developed on one half of a corpus and scored on a half it is never allowed to open. The gap between the two is published as the overfit margin. Section 05. |
| Every attack validated with us switched off | An attack that could not have succeeded with MoorAI removed from the path is an invalid sample: neither a catch nor a miss. Section 06. |
| The model's own contribution subtracted | Every attack goes to the underlying model first. What it refuses unaided is not creditable to us, and is removed before our coverage is stated. Section 07. |
| Reproducible on-device, against a pinned digest | No API key, no scoring service, no network call. One local model, pinned by digest; splits recomputable from the corpus with no stored seed. Section 11. |
This page names the taxonomy those corpora are keyed to, the grading standard we grade ourselves against, the one model we run and what each harness measures.
The argument underneath all of it: a coverage percentage without a corpus, a false-positive rate, a held-out set and a refusal baseline is a number about nothing. We would rather publish a smaller figure that survives inspection than a larger one that does not. Terms used here are defined in the glossary.
The method is ungated. This page covers how corpora are built and labelled, how the locked split works, which baselines a sample has to survive before it counts, what each outcome means, and what makes a run reproducible. The release-specific breakdown behind the headline results is in the full benchmark report.
The corpora and the scoring harness are published as a separate, Apache-2.0 benchmark anyone can run. It holds 299 attack samples and 878 benign, 404 of them hard negatives, across five AMTSO attack vectors, and it scores any product through a small adapter, not just ours. The corpus, the scoring arithmetic and the split algorithm are on the benchmark page; the scored runs are on the leaderboard.
Three external inputs, all of them named
Three external things, and only three: a taxonomy, a grading standard, and a local model. Everything else in the measurement stack is code in the repository, described in section 09.
The published attack-family taxonomy our red-team corpora are keyed to — ten families: CipherChat, FlipAttack, h4rm3l, DAN, AutoDAN, BoN, AdvPrefix, PAP, PAIR and TAP. Every attack sample we score carries the family it belongs to, so coverage can be reported per family rather than as one averaged figure that hides which half of the taxonomy is uncovered. We use the taxonomy (the family definitions and their published techniques) as the labelling scheme for corpora we generate ourselves. Source: AISecurityLab/hackagent ↗. Each family is defined in the glossary.
The Anti-Malware Testing Standards Organization's Guidelines for Testing of Agentic Security Products v1.0 ↗, published 2 September 2026 — the first testing standard written for this product category. We use it as the grading criteria: its six test-case classification dimensions (attack vector, target of protection, environment type, harm type, severity with disclosed criteria, required capability), its rule that model refusal must not be counted as product detection or prevention, and its request for a distribution of outcomes rather than a single pass or fail. AMTSO has not reviewed, certified or endorsed MoorAI — we read their document and graded ourselves against it.
One model, run locally through Ollama: llama3:latest, 8.0B parameters, Q4_0 quantization, digest 365c0bd3c000. It plays two roles: the target whose refusal baseline we measure per attack family, and the judge in the optional semantic escalation path. It is explicitly a local proxy, not a frontier model. Running it locally is deliberate: the corpora never leave the machine, and the measurement costs nothing per run, which is what makes repeated-run variance affordable at all.
All authored in-repo and versioned alongside the code, so any number here can be re-run: a family-keyed HackAgent set, a first held-out set (burned by tuning and retired as a generalisation measure; see discipline), a mutation-generated set split into tune and locked halves, a 610-prompt benign corpus with 269 adversarially-shaped hard negatives, and four dedicated per-vector corpora for the tool/MCP supply chain (72 attacks, 25 benign), memory / cross-agent propagation (42 attacks, 25 benign), outbound action (57 attacks, 24 benign) and indirect content (45 attacks, 17 benign).
What we do not use
Third-party testing tools we did not run, listed by name.
We have not used garak, PyRIT, promptfoo, Burp Suite, Metasploit, or any other third-party penetration-testing or LLM red-teaming tool to produce any number published on this site. Not one figure here came from a scan we ran with somebody else's scanner.
Nor have we been tested by an independent lab. Every measurement below is self-run, on our own corpora, on our own hardware. That is why the harnesses are in a public MIT repository rather than described in a PDF: the falsifiable version of "trust us" is "re-run it".
Six properties a number must pass before we publish it
AMTSO's guidelines are written for testers, not vendors. We grade against them anyway, because the failure mode they exist to prevent is exactly the one a security buyer faces when a vendor publishes its own numbers: a headline percentage with no stated corpus, no stated scope, no false-positive line, and no way to tell which stage of the product was actually exercised. These are the six properties a figure has to satisfy before it goes on this site.
| Property | What it requires |
|---|---|
| Stated scope | Which agent hosts, which surfaces and which stages of the product the number covers — and, named explicitly, which it does not. |
| Representative corpus | Attack samples spread across the ways an agent is actually attacked, rather than concentrated in the vector that is easiest to write samples for. Section 04 is how that spread is constructed. |
| Held-out measurement | A locked corpus that detection logic is never tuned against, kept separate from the corpus used for development. Section 05. |
| False positives on their own line | Recall published alongside benign-sample results, never in place of them, and never averaged together into one flattering score. |
| Baseline attribution | Credit only for what the product adds over what the underlying model already refuses. Section 07. |
| End-to-end exercise | Detection measured through the shipping product, at the entry point a real prompt or tool call takes — not against a detector function called in isolation. |
Corpora authored in the open, labelled against all six AMTSO dimensions
Every sample is authored in-repo and versioned alongside the code. Corpora are constructed, not captured from live incidents — and the labelling scheme is what stops that construction from quietly collapsing onto the samples that are easiest to write.
The development corpus carries the attack and benign samples used while building detection. Detection logic is tuned against it, so by construction it produces no publishable recall number and we do not quote one from it. The locked held-out set — 44 attacks and 10 benign samples — is split off before tuning and never used to develop or adjust a detector; it is the only set that carries a generalisation claim. The benign corpus is 610 samples of legitimate work, 269 of them adversarially-shaped hard negatives. Alongside these sit the family-keyed HackAgent set and the four per-vector corpora listed in section 01.
The per-vector corpora, the family-keyed set, the benign corpus and the scoring harness are republished as a standalone Apache-2.0 benchmark, gitayg/agentic-security-benchmark ↗, so that a competitor can run their own product through the same samples and publish the result without inheriting a copyleft obligation. 299 attack samples and 878 benign, 404 of them hard negatives, with zero dependencies and no model in the loop. The locked test halves stay withheld; only the tune halves ship. What is in it and how to score against it: the benchmark page.
Every classified attack sample is labelled along AMTSO's six test-case classification dimensions. AMTSO's own values are used verbatim; any local extension is prefixed and reported separately, so a locally-invented value can never be mistaken for AMTSO coverage. Labels live in a sidecar file; the corpora themselves are never modified by the labelling pass, so relabelling can never silently change what a sample is.
| Dimension | What it records about a sample |
|---|---|
| Attack vector | How the attack reaches the agent — the six values below. |
| Target of protection | What is being defended: the agent, the user's data, the host, or a downstream system. |
| Environment type | The setting the sample assumes — which agent host, and how the device is configured. |
| Harm type | What goes wrong if it succeeds: data leaves, code executes, state is corrupted, an action is taken. |
| Severity | Recorded against disclosed criteria, so a severity label can be argued with rather than accepted. |
| Required capability | What the attacker must already have for the sample to be realistic — a poisoned page, a published MCP server, a prior session. |
AMTSO defines five attack vectors, the dimension that most determines whether a corpus is representative. The sixth row below is ours, not AMTSO’s: a bucket for samples that map to none of the five, kept visible rather than forced into a vector they do not belong to:
| Vector | What it covers |
|---|---|
| 1 · Direct input | Prompts and instructions supplied by the user, or by an attacker posing as one. |
| 2 · Indirect content | Instructions hidden in fetched pages, files and retrieved documents — the delivery path for indirect prompt injection. |
| 3 · Tool / MCP supply chain | Hostile tool definitions, servers and installers reached over MCP. |
| 4 · Outbound action | The action the agent is being steered into taking. This is the enforcement vector, and the only one whose result depends on how the device is configured, so it is scored per device state rather than as one number. |
| 5 · Memory / cross-agent | Persisted state and content passed between agents — samples that are sequences, written in one session and consumed in a later one. |
| n/a · Static code artifact | Not an AMTSO vector; a local bucket. Instructions embedded in source, configuration and repository files. These samples map to none of the five, and are counted separately rather than assigned to the nearest one, because inflating a vector’s count is how a corpus starts looking more representative than it is. |
The dimension-coverage reporter prints the distribution across all six on every run and flags zero-coverage cells by name rather than reporting an average, because an average is exactly the thing that hides an empty cell.
A benign corpus made of obviously-benign prompts measures nothing: no detector fires on "write me a unit test". So 269 of the 610 benign samples are engineered to look malicious while being ordinary work — legitimate security questions, benign role-play, urgent-sounding routine requests, exfiltration-shaped but innocent prompts, and 101 deliberately obfuscated benign samples (base64 blobs, minified JavaScript, non-English prose, leetspeak identifiers, ASCII tables) added because the previous corpus contained exactly one. They are grouped into twin families: each hard negative is written as the benign twin of an attack family, sharing its surface features and none of its intent. That grouping is what lets a false-positive result say which kind of benign prompt we misfire on rather than only how often — a rate alone tells you nothing you can act on.
Numbers we are not allowed to tune against
A held-out set is a consumable. It measures generalisation exactly once per decision made in ignorance of it, and every look costs some of that.
What this discipline produces: 86.4% recall (38 of 44 attacks) at 100% precision, 0 false positives across the set's 10 benign samples, on a corpus no detector was ever tuned against. It is a smaller number than the tune half returns, and it is the only one of the two that carries a generalisation claim. The per-family split and the overfit margin between the halves are in the benchmark report.
01 Split once, stratified by family and by transformation axis
A corpus is divided a single time into a tune half and a locked test half, stratified so both halves carry the same mix of attack families and the same mix of transformation axes. Stratification matters: an unstratified random split can hand one half every homoglyph sample, and the resulting score then measures which bucket landed where rather than whether the detectors generalise.
02 Deterministic, with no stored seed
Within each stratification bucket, samples are sorted by id and assigned to the two halves alternately. There is no random number generator and therefore no seed to record, lose or quietly change: the split is recomputable from the corpus alone. Anyone with the repository derives the same two halves we did, which is what makes a disagreement about a result checkable rather than a matter of trust.
03 The test half is not read by anyone tuning a detector
The boundary is enforced by file. Detector work opens the tune half; the locked half is not opened during tuning at all. The reason is that tuning against a held-out set silently converts it into a training set, through ordinary iteration rather than cheating. Each time a threshold is nudged, a pattern widened or a wave of rules shipped because the held-out score moved, information from that set has entered the detectors, and the number it produces afterwards describes memorisation rather than generalisation. Nothing about the set looks different once this has happened, which is precisely why it has to be prevented structurally rather than noticed later.
04 A burned set is retired, not reused
We enforce the boundary because we once broke it. An earlier held-out set was tuned against and thereby destroyed as a generalisation measure. There is no way to un-burn it (the information cannot be taken back out of the detectors), so it was retired rather than quietly re-used with a caveat, and a fresh set was generated by a mutation pass that deliberately does not read the detectors. Reusing it would have produced a higher number and a worse measurement.
05 The gap between the halves is itself the finding
Both halves are scored and both are reported. The difference between the tune half's score and the locked half's score is published as the overfit margin, because a method that reports only the locked score discards the one measurement that says how much of the tuning was fitting rather than detecting.
Every attack is verified to work with MoorAI switched off
Two baselines run underneath every coverage figure, and both remove credit rather than adding it. This is the first of them: would this have gone wrong without us? The second — what the model refuses unaided — is section 07.
A test case counts only if the malicious outcome could actually occur with the product removed from the path. An attack sample that resolves to nothing executable, that targets a capability the agent does not have, or that fails for its own reasons is an invalid sample, not a miss, and it is not a catch either. Dropping it costs us both ways, which is the point: a corpus quietly padded with attacks that were never going to succeed inflates recall for free, and blocking something that was never going to happen is not protection. The enforcement side of this is mechanical: before an action is scored as prevented, the harness confirms the action actually executes when the product is not in the path.
We subtract what the model would have caught anyway
The attacks the underlying model refuses on its own are not the ones that reach a working agent. The ones it lets through are, and that remainder is the only region a second layer operates in — so it is the region our coverage figure is stated against. The model is measured first, and what it handles unaided is removed before our number is stated. A detection rate measured without that baseline silently counts the model's refusals as product coverage.
Every attack sample is first put to the underlying model with no product in the path, to establish whether the model refuses it unaided. Samples the model already refuses are excluded from the gap measurement; what remains is the set of attacks that reach a working agent, and it is against that set that the product's contribution is stated. Because a model's response is a sample from a distribution rather than a fact, the baseline is measured over repeated runs per attack family, not once.
Skipping this step is the most common way a published detection rate becomes uninterpretable. A product measured against the full corpus is silently credited with every attack the model would have declined on its own. Since that share is large and varies by family, two products with the same headline recall can add very different amounts of real protection. Running the baseline is uncomfortable, because it can only ever lower your number. Its most useful output for us has been negative: it showed that an entire detection wave we had already shipped bought almost no marginal protection, because the model refused those families anyway. That result produced the priority rule we follow: measure a family's refusal baseline before building detectors for it, and build where the model complies.
Run that way against the local Llama 3 8B baseline, the number left is 95.2% gap coverage: of the 21 attacks the model does not refuse unaided, MoorAI catches 20, a marginal 45.5% (20/44) of the corpus. Run against the deployed Claude Sonnet assistant, which refuses 75.8% of the attacks it receives (25 of 33) and whose provider rejects another 11 of 44 before the model sees them, the gap is 8 attacks, MoorAI catches 6 (75.0%), and the corpus-wide marginal is 13.6% (6/44). The frontier figure is lower because a frontier assistant refuses more on its own, which leaves less for MoorAI to add. Residual exposure, the cases no layer stops, is 1 of 44 (2.3%) against the 8B and 2 of 44 (4.5%) against Sonnet. Gap coverage is the figure that describes what the product adds: it is stated against the gap, not against the full corpus. Per-family gap coverage and the refusal rates it was computed from are in the full benchmark report.
Ask what the underlying model refuses on its own, and what the vendor's number looks like once those attacks are taken out. A recall figure measured against the full attack corpus credits the product with catches the model would have refused unaided, which is why two products with identical recall can add wildly different amounts of protection. The number that describes what a product adds is coverage of the gap: the attacks the model does not refuse on its own. A figure published without that baseline is unattributed — it does not say how much of the protection came from the product and how much came from the model underneath it. Measuring the gap is the single methodological choice on this page we would most like other vendors to copy, because it is the one that makes published detection rates comparable at all.
The repeatable refusal baseline is measured against llama3:latest (8.0B, Q4_0), because a baseline that costs money per sample cannot be run repeatedly and a baseline run once is not a baseline. The same probes are also run, five times per sample, against the deployed Claude Sonnet assistant through the claude CLI, which carries Anthropic's own defences. A frontier assistant refuses more, so the 8B baseline leaves a larger gap and a more flattering marginal figure: 45.5% of the corpus against the 8B, 13.6% against Sonnet. Both are published; the Sonnet figure is the one a buyer running that assistant should expect.
Detection and prevention are never the same number
A sample is fed to the shipping agent through the same entry point a real prompt or tool call takes. Detector functions are not scored in isolation, because a detector that returns a finding no deployment can see is not a detection. That lesson produced the cross-agent end-to-end reachability scorer.
| Term | Definition used throughout |
|---|---|
| Attack sample | One prompt, tool call, file or piece of retrieved content constructed to make an agent act against its user. |
| Benign sample | Legitimate developer or security work that must not be flagged. A hard negative is a benign sample deliberately shaped to resemble an attack. |
| Detection | A categorized finding at or above the alert threshold. |
| Prevention | An action the agent was stopped from executing. Only a sample that carries an action can be prevented. |
| Right reason | A catch credited to a detector that matches the sample's actual attack class, rather than a coincidental match by an unrelated rule. |
| False positive | A benign sample producing any finding at or above the alert threshold — regardless of category, because a wrong-category alert is still an alert the user has to deal with. |
| Enforcing policy | A device enrolled to a tenant whose policy sets the relevant category to block, as distinct from alert. |
Three stages of the product are exercised end to end: the tool stage (a tool call before it executes), the index stage (content as it enters the agent's context) and the output stage (what the agent produces). Which stage a sample can be reached at is a property of the sample, not a choice — supply-chain samples are tool metadata and config files reachable only at the tool, file and index stages, and cross-agent samples may be event-graph shapes with no text at all.
The outcome vocabulary
AMTSO asks for a distribution of outcomes rather than a single pass or fail, and these are the buckets a scored sample lands in. Two of them are not creditable to the product at all, and the vocabulary exists to keep them out of its score.
| Outcome | What it means, and who gets the credit |
|---|---|
| Prevented | The action was stopped before it executed. Requires an enforcing policy and a sample that carries an action; jailbreak and injection material has nothing to stop, so for that material the correct outcome is detection, not prevention. |
| Detected, not prevented | Flagged at or above the alert threshold, but nothing was stopped — either because the sample carries no action, or because policy is set to alert rather than block. A real result, and a weaker claim than prevention. The two are never merged into one blended "block rate". |
| Model refusal | The model declined unaided, with the product out of the path. Not creditable to the product — AMTSO states this explicitly, and it is the rule that the refusal baseline in section 07 exists to operationalise. |
| Model recognition | The model identified the attempt as an attack without the product in the path, whether or not it went on to refuse. Like refusal, this belongs to the model and is reported separately rather than folded into product coverage. |
| Missed | No finding at or above the alert threshold, and no refusal. The bucket whose contents are most worth reading. |
| Inconclusive | The run does not adjudicate — the sample could not be shown to work with the product absent, the harness errored, or repeated runs disagree. Counted as its own outcome and never quietly redistributed into either success or failure. |
A local refinement sits on top of this: a catch is additionally scored as right-reason or not, and the right-reason rate is reported separately from recall wherever it is materially below it. An incidental catch — the sample flagged by a detector aimed at something else — is a real detection, but the categorization a security team sees is wrong, and it is a much weaker guarantee of generalisation, because it does not survive the attacker changing the one thing the unrelated rule happened to match. The full distribution of outcomes for a specific release, including the right-reason breakdown, is published in the benchmark report.
Eleven harnesses, every test seen failing first
Eleven measurement scripts in the agent repository. Each one is described by what it measures, because a harness that cannot state what would falsify it is not a measurement. All of them import the shipped detection engine rather than re-implementing it, and none of them mutates a detector or a corpus.
| Harness | What it measures |
|---|---|
| Red-team coverage evalscripts/redteam-eval.mjs | Detection coverage across the ten HackAgent families, deterministically and LLM-free: every sample carries a ground-truth shouldDetect label, so coverage, precision and recall are computed by comparing the engine's verdict to that label — no model, no judge, reproducible on any machine. An optional flag routes samples through the on-device model layer instead; the default run never imports it, so the headline number is never model-assisted. Current release: 101/102 deterministic coverage, and 100% (29/29) on this harness's held-out slice at 94% precision (4 false positives / 178). |
| Locked tune/test splitterscripts/split-heldout-v2.mjs | Splits a corpus once into a tune half and a locked test half, stratified by family and transformation axis so both halves carry the same mix; an unstratified split could hand one half every homoglyph sample and make the result meaningless. Within each bucket samples are sorted by id and assigned alternately, so the split is reproducible from the corpus alone with no stored seed. Tuning work may see only the tune half. |
| Adversarial mutation generatorscripts/mutate-attacks.mjs | Generates a fresh attack set by transforming each family's published technique along explicit axes — synonym and voice swaps, directive-in-code-comment / JSON / blockquote indirection, homoglyphs, letter-spacing, Caesar and ROT13, persona and persuasion recombinations. It deliberately does not read the detectors: steering mutations toward what the engine already catches would burn the set. Every transform except a bounded single-character typo axis is information-preserving and reversible, so the malicious core is recoverable and the sample is provably still an attack. |
| Closed-loop adversaryscripts/closed-loop-mutate.mjs | Models an adversary who learns: round 0 is the plaintext cores, round 1 every single-step transformation, and round N+1 every one-step extension of only the chains that evaded in round N. A one-shot generator understates risk because it never composes two evasions; this spends its budget only on productive directions and reports which transformation chains survive. Validity is checked mechanically: every candidate is rendered backwards and must reproduce the malicious core character for character, or it is dropped before it is ever scored. |
| Benign false-positive scorerscripts/score-benign-v2.mjs | The other half of the question recall never answers: across 610 realistic benign developer and agent prompts, 269 of them adversarially shaped hard negatives, how often does the engine fire when it should stay silent? Current release: 3.32% (20/602), specificity 96.68%. Reports the false-positive rate per bucket and per hard-negative twin family, and for every false positive the detector and threat ids that fired. It reuses the same reducer as the recall benchmarks, so the two numbers are apples-to-apples. |
| Model-refusal baselinescripts/measure-refusal-baseline.mjs | Per attack family, how often the model refuses on its own with no product involved — run N times because a single run is a point estimate. This is what converts a raw coverage figure into marginal value: recall added where the model already refuses is worth close to nothing. Its most useful output has been negative — it showed that an entire detection wave we had shipped bought almost no marginal protection, because the model refused those families anyway. |
| Variance harnessscripts/measure-variance.mjs | Repeated runs of the same scored set, reporting the distribution and per-sample stability rather than one number. It exists because we watched model-assisted figures move between runs — a judge recovering eight attacks once and five the next time, a benign false-positive count drifting by one. AMTSO asks for the distribution of outcomes rather than hiding variability behind a single pass or fail; this produces it. Measurement only: it mutates no detector, no corpus and no other script. |
| Action-layer blocking validatorscripts/moorai-validate-blocking.mjs | The one that tests prevention rather than detection. It assumes the agent has already lost (hijacked, jailbroken, whatever) and asks whether the malicious tool call it then attempts is denied. For each action it feeds a real PreToolUse payload to the real hook as a subprocess and records the actual decision from its stdout; it does not re-implement the hook's logic, which is the entire point. Current release: 12 of 12 malicious tool calls stopped under an enforcing policy. This is the harness behind any claim that uses the word "blocked". |
| Per-vector scorersscripts/score-vectors.mjs | Scores two AMTSO attack vectors that have detectors and need their own measurement: tool / skill / extension / MCP supply chain and memory, context and cross-agent propagation. They need their own scorer because neither fits "one sample = one string": supply-chain samples are tool metadata and config files only reachable at the tool, file and index stages, and cross-agent samples are sequences (written in one session, consumed in a later one) or event-graph shapes with no text at all. Optimistic and strict readings are reported separately and are not interchangeable. |
| Cross-agent end-to-end reachabilityscripts/score-vector5-production.mjs | Whether a real deployment can actually see the cross-agent detections, as opposed to whether the library function returns a finding. It refuses to call the library at all: it seeds the on-device event window and then drives the two paths a deployment really has (the real PreToolUse hook and the real agentwatch emitter), counting only what arrives at a stand-in management server. It was written because the library scored full marks on a capability nothing on the enforcement path was calling. |
| AMTSO dimension-coverage reporterscripts/amtso-coverage.mjs | Classifies every corpus sample along AMTSO's six dimensions and reports the distribution, flagging zero-coverage cells by name rather than reporting an average. Labels live in a sidecar; the corpora are never modified. AMTSO's own values are used verbatim and any local extension is prefixed and reported separately so it can never be mistaken for AMTSO coverage. It finds the holes in our testing. |
Five rules, each of which cost us a number
A scoring script is easy to write. These rules are what make its output worth reading.
01 Locked held-out splits the tuning work never sees
A corpus is split once, stratified, and the test half is off-limits to detector work: it is a separate file that the tuning wave does not open. We enforce this because we broke it once: an earlier held-out set was tuned against and thereby destroyed as a generalisation measure, and it has been retired rather than quietly re-used. The gap between the tune half's score and the locked half's score is published as the overfit margin, because that gap is the finding.
02 Baseline validation — a sample only counts if the attack works with the product off
Blocking something that was never going to succeed is not protection. Two baselines run underneath the coverage numbers: the refusal baseline, which removes credit for anything the model would have refused anyway, and the enforcement baseline, which confirms an action actually executes when the product is not in the path. AMTSO states the refusal principle; the practical consequence is a priority rule we follow: measure the refusal baseline for a family first, and build only where the model complies.
03 False positives measured alongside, reported separately
Every recall figure has a precision figure measured on the same engine against the 610-prompt benign corpus, and the two are never combined into a single flattering score. The corpus is deliberately hostile to us: 269 of its 610 samples are hard negatives: legitimate security questions, benign role-play, urgent-sounding ordinary requests, exfiltration-shaped but innocent prompts, and 101 deliberately obfuscated benign samples (base64 blobs, minified JavaScript, non-English prose, leetspeak identifiers, ASCII tables) added because the previous corpus contained exactly one. They are grouped into twin families so we can see which kind of benign prompt we misfire on, not just how often.
04 Falsify first — every test seen failing before it is trusted
A test that has never been observed failing is not evidence; it is a green light with no wiring behind it. So the production path is deliberately broken, the test is confirmed to go red, the break is reverted, and what was broken is recorded. This rule catches the embarrassing class of bug: the harness that scores a capability nothing in the shipping product calls, which is exactly what the cross-agent end-to-end reachability harness was written to expose after a library-level score of full marks turned out to be unreachable in a real deployment.
The clearest example of why mutation is the only version of this that works is a benign fixture that produced zero findings at every stage. It carried an assertion that its clean result must not silently mean "never scanned", and that assertion was green. It would also have been green on the day the scan was deleted, because from outside the harness zero findings and no scan are the same observation. Only breaking the production line — removing the scan and confirming the test goes red — distinguishes them, and until that is done the assertion is decoration. Every claim on this page has a test that has been seen red.
05 Repeated runs, variance reported instead of a lucky pass
Anything involving a model is a sample from a distribution. A single passing run is a point estimate presented as a fact, and we have watched those estimates move. That is why model-assisted figures are run N times and published as a spread, and why the deterministic, LLM-free number is always the headline while the model-assisted one is reported separately and never merged into it.
Reproducible on any machine, against a pinned model digest
"Re-run it and disagree" is the only offer we can make in place of an independent lab, so it has to work. Four properties make it work.
01 The headline path is deterministic and LLM-free
Every sample carries a ground-truth shouldDetect label, and the default scoring run compares the engine's verdict to that label with no model anywhere in the loop. Coverage, precision and recall from that path are byte-identical on any machine. The model-assisted path exists behind an explicit flag; the default run never imports it, so a headline number is never quietly model-assisted.
02 Splits are fixed and derivable, not seeded
The tune/test split is recomputed from the corpus by sorting and alternating rather than sampling, so there is no seed to publish and no seed to get wrong. Corpus generation is seeded and fixed. Reproducing our halves requires the repository and nothing else — see section 05.
03 The model is pinned by digest
Where a model is involved at all — the refusal baseline and the semantic escalation judge — it is one pinned local build: llama3:latest, 8.0B parameters, Q4_0, digest 365c0bd3c000. A tag alone would not be reproducible, because a tag moves; the digest is what makes "the same model" checkable a year from now.
04 Everything runs on-device, so anyone can run it
There is no API key, no scoring service and no network call anywhere in the measurement path. Attack corpora are never sent to a third party, results are never adjudicated by a remote judge, and the cost of a run is electricity, which is what makes repeated runs, and therefore variance reporting, affordable at all. Model-assisted figures are run N times and published as a spread, never as a lucky single pass and never merged into the deterministic number. The agent, the harnesses and the corpora are MIT.
The scope every figure above is measured within:
| Scope item | What is covered |
|---|---|
| Architecture | On-device. Detection runs locally; a category, a risk level and a one-way hash (plus the login and hostname, which the console pseudonymises on arrival) are the only things that leave the machine. Content-free by construction, which is also why no corpus can be built from user telemetry. |
| Stages measured | Tool stage, index stage and output stage, each exercised through the shipping product end to end rather than at the library boundary. |
| Hosts covered | Host-level hooks, plus MCP enforcement that is host-independent — it applies wherever an agent reaches a tool over MCP. Not every agent host is registered; the release-specific list is in the benchmark report. |
| Enforcement scope | Prevention claims apply to a device enrolled to a tenant with an enforcing policy. An unenrolled device enforces no tenant policy and blocks no outbound action. That is by design. |
| Corpora | A family-keyed HackAgent set, a mutation-generated set split into tune and locked halves, a locked held-out set of 44 attacks and 10 benign samples, a 610-sample benign corpus with 269 hard negatives, and four per-vector corpora. |
| Independent review | None. Every figure is self-measured, on self-authored corpora, on our own hardware. |
Self-run, and not certified
Every figure here is self-run, on self-authored corpora, on our own hardware. No third-party lab has verified, reproduced or audited any of it. What we offer in place of a certificate is a repository: the agent, the harnesses and the corpora are open source under MIT, so the falsifiable version of this page is somebody re-running it and disagreeing.
We read AMTSO's published Guidelines for Testing of Agentic Security Products v1.0 and graded our own testing against its criteria. That is self-assessment, not accreditation. AMTSO has not reviewed our methodology, has not tested our product, has not certified anything, and has no relationship with us. Nothing on this page is an AMTSO result, an AMTSO score, or an AMTSO endorsement, and any reading of it as one is wrong. The same applies to OWASP, MITRE and every other body named here.
Run your own tool through this, and grade us the same way
The method above is the whole method. Nothing is held back for a sales call, and checking it does not depend on our word.
The agent, the eleven harnesses and every corpus are in the open-source repository under MIT. The split recomputes from the corpus with no stored seed, the headline path is deterministic and LLM-free, and the model is pinned by digest — so a run on your machine and a run on ours are the same run, and a disagreement about a result is checkable rather than a matter of trust. Point the same harnesses at your own guard: label the samples along the same six dimensions, validate that each attack works with the product removed, subtract your model's refusal baseline, and publish the distribution rather than a headline. Grade us by the same standard, and tell us where a number here does not survive it.
The method and the headline results are both on this page, ungated. The full breakdown for a specific release — the outcome distribution, per-family and per-vector coverage, per-device-state enforcement, the false-positive breakdown by twin family and the overfit margin — is in the full benchmark report.
This page describes the method and is deliberately ungated; the headline figures are for the current release and the full breakdown behind them is published in the full benchmark report. Harness paths refer to the open-source MoorAI agent repository (MIT). Corpus sizes and the AMTSO dimension distribution above were produced by running the in-repo reporter; they move as corpora grow, so treat them as of the current release rather than as fixed constants. AMTSO has not reviewed, certified or endorsed MoorAI, and the Guidelines are cited as published criteria we graded ourselves against. OWASP® is a trademark of the OWASP Foundation; ATT&CK® and ATLAS™ are trademarks of The MITRE Corporation; AMTSO is a trademark of the Anti-Malware Testing Standards Organization. Llama 3 is used under the Meta Llama 3 Community License. No third-party penetration-testing or LLM red-teaming tool was used to produce any figure referenced on this page.
Frequently asked questions
Has AMTSO certified or endorsed MoorAI?
No. AMTSO has not reviewed, certified or endorsed MoorAI. We read AMTSO's published Guidelines for Testing of Agentic Security Products v1.0 (2 September 2026) and graded our own testing against its criteria. That is self-assessment, not accreditation, and nothing published is an AMTSO result or score.
Which third-party security testing tools does MoorAI use?
None. No third-party penetration-testing or LLM red-teaming tool — not garak, PyRIT, promptfoo, Burp Suite or Metasploit — was used to produce any published figure. The external inputs are the HackAgent attack taxonomy, AMTSO's testing guidelines as grading criteria, and a local Ollama llama3:latest model (8.0B, Q4_0) used as a refusal baseline and semantic judge. All measurement harnesses are in the open-source agent repository.
How does MoorAI avoid overfitting its detection numbers?
Corpora are split once into a tune half and a locked test half, stratified by attack family and transformation axis, and detector work may only see the tune half. The gap between the two halves is published as the overfit margin. A fresh adversarial mutation generator produces out-of-sample attacks without reading the detectors, and a closed-loop adversary re-mutates only the samples that evaded in the previous round.
Why does MoorAI measure a model-refusal baseline?
Because a guard for AI agents sits behind a model that already refuses many obvious attacks, so a recall figure measured against a full attack corpus silently credits the product with catches the model would have refused unaided. Every attack sample is first put to the underlying model with no product in the path, over repeated runs per attack family. Samples the model already refuses are excluded, and the product's contribution is stated against the remaining gap. Model refusal and model recognition are never counted as product detection.
What makes an attack sample valid under the AMTSO guidelines?
A test case counts only if the malicious outcome could actually occur with the product removed from the path. An attack that resolves to nothing executable, or that targets a capability the agent does not have, is an invalid sample rather than a miss — and equally not a catch. On the enforcement side, the harness confirms an action actually executes with the product out of the path before that action can be scored as prevented.
What are the possible outcomes when MoorAI scores a sample?
Six: prevented (the action was stopped before it executed), detected but not prevented, model refusal, model recognition, missed, and inconclusive. Model refusal and model recognition belong to the underlying model and are not creditable to the product. A benign sample that produces any finding at or above the alert threshold is a false positive regardless of category. Catches are additionally scored as right-reason or not, and the right-reason rate is reported separately from recall wherever it is materially below it.