How we test
We grade MoorAI against AMTSO's Guidelines for Testing of Agentic Security Products v1.0 — on a locked corpus we are not allowed to tune against, with the underlying model's own refusals measured and subtracted, so what is left is what the product adds. Every detection number on this site comes from a harness in the open-source agent repository, and the method that produces it is on this page in full.
What the method produces. Five headline figures for the current release, each with the condition it was measured under. The working behind every one of them — the outcome distribution, the per-family and per-vector breakdown, the per-device-state enforcement table — is in the full benchmark report.
| Result | What it was measured against |
|---|---|
| 95.2% of the model-refusal gap | Of the 21 attacks the underlying model does not refuse unaided, MoorAI catches 20, leaving residual exposure of 1 of 44 (2.3%) that neither layer stops. Counting the model's own refusals alongside it, 97.7% combined. This is the number the refusal baseline exists to produce. Breakdown ↗ |
| 86.4% on the locked held-out set | 38 of 44 attacks on a corpus detector work is never allowed to open, at 100% precision — 0 false positives across its 10 benign samples. Section 05 is why that set is worth more than a larger score from the tune half. Breakdown ↗ |
| 3.32% false positives on benign traffic | 20 of 602, specificity 96.68% — measured on the 610-prompt benign corpus, 269 of it adversarially-shaped hard negatives written to look like the attacks they sit beside. Breakdown ↗ |
| 12 of 12 malicious tool calls stopped | Prevention, not detection: with the agent assumed already hijacked and an enforcing policy in place, every one of the twelve malicious actions it then attempted was denied by the real hook. Scoped to one agent host — see limits. Breakdown ↗ |
| 101/102 deterministic red-team coverage | Across the ten HackAgent families, with no model anywhere in the loop — byte-identical on any machine. The redteam-eval held-out run scores 100% (29/29) at 94% precision (4 false positives / 178). Breakdown ↗ |
| What this page commits us to | What it means |
|---|---|
| A locked split we never tune against | Detection logic is developed on one half of a corpus and scored on a half it is never allowed to open. The gap between the two is published as the overfit margin. Section 05. |
| Every attack validated with us switched off | An attack that could not have succeeded with MoorAI removed from the path is an invalid sample — not a catch, and not a miss. Section 06. |
| The model's own contribution subtracted | Every attack goes to the underlying model first. What it refuses unaided is not creditable to us, and is removed before our coverage is stated. Section 07. |
| Reproducible on-device, against a pinned digest | No API key, no scoring service, no network call. One local model, pinned by digest; splits recomputable from the corpus with no stored seed. Section 11. |
This page names the taxonomy those corpora are keyed to, the grading standard we grade ourselves against, the one model we run, each harness and what it actually measures — and then the limits, because a methodology page that only lists strengths is marketing with footnotes.
The argument underneath all of it: a coverage percentage without a corpus, a false-positive rate, a held-out set and a refusal baseline is a number about nothing. We would rather publish a smaller figure that survives inspection than a larger one that does not. Terms used here are defined in the glossary.
The method is published in full and is not gated. Everything needed to reproduce the approach is here: how corpora are built and labelled, how the locked split works, which baselines a sample has to survive before it counts at all, what each outcome means, and what makes a run reproducible. The headline results are above; the full breakdown behind them for a specific release — the outcome distribution, per-family and per-vector coverage, and the per-device-state enforcement table — is in the full benchmark report.
The corpora and the scoring harness are published as a separate, Apache-2.0 benchmark anyone can run. It holds 286 attack samples and 875 benign, 401 of them hard negatives, across five AMTSO attack vectors — and it scores any product behind a small adapter, not just ours. The corpus, the scoring arithmetic and the split algorithm are on the benchmark page; the scored runs are on the leaderboard.
Three external inputs, all of them named
Three external things, and only three: a taxonomy, a grading standard, and a local model. Everything else in the measurement stack is code in the repository, described in section 09.
The published attack-family taxonomy our red-team corpora are keyed to — ten families: CipherChat, FlipAttack, h4rm3l, DAN, AutoDAN, BoN, AdvPrefix, PAP, PAIR and TAP. Every attack sample we score carries the family it belongs to, so coverage can be reported per family rather than as one averaged figure that hides which half of the taxonomy is uncovered. We use the taxonomy — the family definitions and their published techniques — as the labelling scheme for corpora we generate ourselves. Source: AISecurityLab/hackagent ↗. Each family is defined in the glossary.
The Anti-Malware Testing Standards Organization's Guidelines for Testing of Agentic Security Products v1.0 ↗, published 2 September 2026 — the first testing standard written for this product category. We use it as the grading criteria: its six test-case classification dimensions (attack vector, target of protection, environment type, harm type, severity with disclosed criteria, required capability), its rule that model refusal must not be counted as product detection or prevention, and its request for a distribution of outcomes rather than a single pass or fail. AMTSO has not reviewed, certified or endorsed MoorAI — we read their document and graded ourselves against it.
One model, run locally through Ollama: llama3:latest — 8.0B parameters, Q4_0 quantization, digest 365c0bd3c000. It plays two roles: the target whose refusal baseline we measure per attack family, and the judge in the optional semantic escalation path. It is explicitly a local proxy, not a frontier model, and this matters for reading our results — see limits. Running it locally is deliberate: the corpora never leave the machine, and the measurement costs nothing per run, which is what makes repeated-run variance affordable at all.
All authored in-repo and versioned alongside the code, so any number here can be re-run: a family-keyed HackAgent set, a first held-out set (now burned by tuning and retired as a generalisation measure — see discipline), a mutation-generated set split into tune and locked halves, a 610-prompt benign corpus with 269 adversarially-shaped hard negatives, and four dedicated per-vector corpora for the tool/MCP supply chain (72 attacks, 25 benign), memory / cross-agent propagation (42 attacks, 25 benign), outbound action (57 attacks, 24 benign) and indirect content (45 attacks, 17 benign).
What we do not use
Listing tools you did not run is the whole point of a methodology page, so here it is explicitly rather than by omission.
We have not used garak, PyRIT, promptfoo, Burp Suite, Metasploit, or any other third-party penetration-testing or LLM red-teaming tool to produce any number published on this site. Not one figure here came from a scan we ran with somebody else's scanner. Where a well-known tool name would make a claim sound more credible, its absence from our stack is the honest fact, and we would rather say so than let a logo imply a test that never happened.
Nor have we been tested by an independent lab. Every measurement below is self-run, on our own corpora, on our own hardware. That is a real limitation, not a formality — see limits — and it is why the harnesses are in a public MIT repository rather than described in a PDF: the falsifiable version of "trust us" is "re-run it".
Seven properties a number must pass before we publish it
AMTSO's guidelines are written for testers, not vendors. We grade against them anyway, because the failure mode they exist to prevent is exactly the one a security buyer faces when a vendor publishes its own numbers: a headline percentage with no stated corpus, no stated scope, no false-positive line, and no way to tell which stage of the product was actually exercised. These are the seven properties a figure has to satisfy before it goes on this site.
| Property | What it requires |
|---|---|
| Stated scope | Which agent hosts, which surfaces and which stages of the product the number covers — and, named explicitly, which it does not. |
| Representative corpus | Attack samples spread across the ways an agent is actually attacked, rather than concentrated in the vector that is easiest to write samples for. Section 04 is how that spread is constructed and section 12 is where it is still uneven. |
| Held-out measurement | A locked corpus that detection logic is never tuned against, kept separate from the corpus used for development. Section 05. |
| False positives on their own line | Recall published alongside benign-sample results, never in place of them, and never averaged together into one flattering score. |
| Baseline attribution | Credit only for what the product adds over what the underlying model already refuses. Section 07. |
| End-to-end exercise | Detection measured through the shipping product, at the entry point a real prompt or tool call takes — not against a detector function called in isolation. |
| Stated limitations | Every surface that is not covered, named. Section 12. |
Corpora authored in the open, labelled against all six AMTSO dimensions
Every sample is authored in-repo and versioned alongside the code. Corpora are constructed, not captured from live incidents — a real limitation, stated again in section 12 — and the labelling scheme is what stops that construction from quietly collapsing onto the samples that are easiest to write.
The development corpus carries the attack and benign samples used while building detection. Detection logic is tuned against it, so by construction it produces no publishable recall number and we do not quote one from it. The locked held-out set — 44 attacks and 10 benign samples — is split off before tuning and never used to develop or adjust a detector; it is the only set that carries a generalisation claim. The benign corpus is 610 samples of legitimate work, 269 of them adversarially-shaped hard negatives. Alongside these sit the family-keyed HackAgent set and the four per-vector corpora listed in section 01.
The per-vector corpora, the family-keyed set, the benign corpus and the scoring harness are republished as a standalone Apache-2.0 benchmark — gitayg/agentic-security-benchmark ↗ — deliberately, so a competitor can run their own product through the same samples and publish the result without inheriting a copyleft obligation. 286 attack samples and 875 benign, 401 of them hard negatives, with zero dependencies and no model in the loop. The locked test halves stay withheld; only the tune halves ship. What is in it and how to score against it: the benchmark page.
Every classified attack sample is labelled along AMTSO's six test-case classification dimensions. AMTSO's own values are used verbatim; any local extension is prefixed and reported separately, so a locally-invented value can never be mistaken for AMTSO coverage. Labels live in a sidecar file — the corpora themselves are never modified by the labelling pass, so relabelling can never silently change what a sample is.
| Dimension | What it records about a sample |
|---|---|
| Attack vector | How the attack reaches the agent — the six values below. |
| Target of protection | What is being defended: the agent, the user's data, the host, or a downstream system. |
| Environment type | The setting the sample assumes — which agent host, and how the device is configured. |
| Harm type | What goes wrong if it succeeds: data leaves, code executes, state is corrupted, an action is taken. |
| Severity | Recorded against disclosed criteria, so a severity label can be argued with rather than accepted. |
| Required capability | What the attacker must already have for the sample to be realistic — a poisoned page, a published MCP server, a prior session. |
AMTSO defines five attack vectors, the dimension that most determines whether a corpus is representative. The sixth row below is ours, not AMTSO’s — a bucket for samples that map to none of the five, kept visible rather than forced into a vector they do not belong to:
| Vector | What it covers |
|---|---|
| 1 · Direct input | Prompts and instructions supplied by the user, or by an attacker posing as one. |
| 2 · Indirect content | Instructions hidden in fetched pages, files and retrieved documents — the delivery path for indirect prompt injection. |
| 3 · Tool / MCP supply chain | Hostile tool definitions, servers and installers reached over MCP. |
| 4 · Outbound action | The action the agent is being steered into taking. This is the enforcement vector, and the only one whose result depends on how the device is configured, so it is scored per device state rather than as one number. |
| 5 · Memory / cross-agent | Persisted state and content passed between agents — samples that are sequences, written in one session and consumed in a later one. |
| n/a · Static code artifact | Not an AMTSO vector — a local bucket. Instructions embedded in source, configuration and repository files. These samples map to none of the five, and are counted separately rather than assigned to the nearest one, because inflating a vector’s count is how a corpus starts looking more representative than it is. |
The dimension-coverage reporter prints the distribution across all six on every run and flags zero-coverage cells by name rather than reporting an average, because an average is exactly the thing that hides an empty cell. The current distribution across the classified attack samples is in limits, where it belongs, since its interest is where it is still lopsided.
A benign corpus made of obviously-benign prompts measures nothing: no detector fires on "write me a unit test". So 269 of the 610 benign samples are engineered to look malicious while being ordinary work — legitimate security questions, benign role-play, urgent-sounding routine requests, exfiltration-shaped but innocent prompts, and 101 deliberately obfuscated benign samples (base64 blobs, minified JavaScript, non-English prose, leetspeak identifiers, ASCII tables) added because the previous corpus contained exactly one. They are grouped into twin families: each hard negative is written as the benign twin of an attack family, sharing its surface features and none of its intent. That grouping is what lets a false-positive result say which kind of benign prompt we misfire on rather than only how often — a rate alone tells you nothing you can act on.
Numbers we are not allowed to tune against
A held-out set is a consumable. It measures generalisation exactly once per decision made in ignorance of it, and every look costs some of that.
What this discipline produces: 86.4% recall — 38 of 44 attacks — at 100% precision, 0 false positives across the set's 10 benign samples, on a corpus no detector was ever tuned against. It is a smaller number than the tune half returns, and it is the only one of the two that carries a generalisation claim. The per-family split and the overfit margin between the halves are in the benchmark report.
01 Split once, stratified by family and by transformation axis
A corpus is divided a single time into a tune half and a locked test half, stratified so both halves carry the same mix of attack families and the same mix of transformation axes. Stratification is not decoration: an unstratified random split can hand one half every homoglyph sample, and the resulting score then measures which bucket landed where rather than whether the detectors generalise.
02 Deterministic, with no stored seed
Within each stratification bucket, samples are sorted by id and assigned to the two halves alternately. There is no random number generator and therefore no seed to record, lose or quietly change: the split is recomputable from the corpus alone. Anyone with the repository derives the same two halves we did, which is what makes a disagreement about a result checkable rather than a matter of trust.
03 The test half is not read by anyone tuning a detector
This is a file boundary, not an intention. Detector work opens the tune half; the locked half is not opened during tuning at all. The reason is that tuning against a held-out set silently converts it into a training set — not through cheating but through ordinary iteration. Each time a threshold is nudged, a pattern widened or a wave of rules shipped because the held-out score moved, information from that set has entered the detectors, and the number it produces afterwards describes memorisation rather than generalisation. Nothing about the set looks different once this has happened, which is precisely why it has to be prevented structurally rather than noticed later.
04 A burned set is retired, not reused
We enforce the boundary because we once broke it. An earlier held-out set was tuned against and thereby destroyed as a generalisation measure. There is no way to un-burn it — the information cannot be taken back out of the detectors — so it was retired rather than quietly re-used with a caveat, and a fresh set was generated by a mutation pass that deliberately does not read the detectors. Reusing it would have produced a higher number and a worse measurement.
05 The gap between the halves is itself the finding
Both halves are scored and both are reported. The difference between the tune half's score and the locked half's score is published as the overfit margin, because a method that reports only the locked score discards the one measurement that says how much of the tuning was fitting rather than detecting.
Every attack is verified to work with MoorAI switched off
Two baselines run underneath every coverage figure, and both remove credit rather than adding it. This is the first of them: would this have gone wrong without us? The second — what the model refuses unaided — is section 07.
A test case counts only if the malicious outcome could actually occur with the product removed from the path. An attack sample that resolves to nothing executable, that targets a capability the agent does not have, or that fails for its own reasons is an invalid sample, not a miss — and it is equally not a catch. Dropping it costs us both ways, which is the point: a corpus quietly padded with attacks that were never going to succeed inflates recall for free, and blocking something that was never going to happen is not protection. The enforcement side of this is mechanical: before an action is scored as prevented, the harness confirms the action actually executes when the product is not in the path.
We subtract what the model would have caught anyway
The attacks the underlying model refuses on its own are not the ones that reach a working agent. The ones it lets through are, and that remainder is the only region a second layer operates in — so it is the region our coverage figure is stated against. The model is measured first, and what it handles unaided is removed before our number is stated. A detection rate measured without that baseline silently counts the model's refusals as product coverage.
Every attack sample is first put to the underlying model with no product in the path, to establish whether the model refuses it unaided. Samples the model already refuses are excluded from the gap measurement; what remains is the set of attacks that reach a working agent, and it is against that set that the product's contribution is stated. Because a model's response is a sample from a distribution rather than a fact, the baseline is measured over repeated runs per attack family, not once.
Skipping this step is the most common way a published detection rate becomes uninterpretable. A product measured against the full corpus is silently credited with every attack the model would have declined on its own — and since that share is large and varies by family, two products with the same headline recall can add very different amounts of real protection. Running the baseline is uncomfortable, because it can only ever lower your number. Its most useful output for us has been negative: it showed that an entire detection wave we had already shipped bought almost no marginal protection, because the model refused those families anyway. That result produced the priority rule we now follow — measure a family's refusal baseline before building detectors for it, and build where the model complies.
Run that way, the number the baseline leaves us with is 95.2% gap coverage: of the 21 attacks the model does not refuse unaided, MoorAI catches 20, and 1 of 44 (2.3%) is left as residual exposure — the case no layer stops. Counting the model's own refusals alongside product detections, 97.7% combined. The 95.2% is the figure that describes what the product adds — it is stated against the gap, not against the full corpus, and it is smaller than a full-corpus figure would have been. Per-family gap coverage and the refusal rates it was computed from are in the full benchmark report.
Ask what the underlying model refuses on its own, and what the vendor's number looks like once those attacks are taken out. A recall figure measured against the full attack corpus credits the product with catches the model would have refused unaided — which is why two products with identical recall can add wildly different amounts of protection. The number that describes what a product adds is coverage of the gap: the attacks the model does not refuse on its own. A figure published without that baseline is unattributed — it does not say how much of the protection came from the product and how much came from the model underneath it. Measuring the gap is the single methodological choice on this page we would most like other vendors to copy, because it is the one that makes published detection rates comparable at all.
The refusal baseline is measured against llama3:latest (8.0B, Q4_0), because a baseline that costs money per sample cannot be run repeatedly and a baseline run once is not a baseline. A frontier model refuses more, so measuring against an 8B model produces a larger apparent gap and therefore a more flattering marginal figure than a frontier baseline would. We state this rather than banking it — see limits, where the specific mechanism is named.
Detection and prevention are never the same number
A sample is fed to the shipping agent through the same entry point a real prompt or tool call takes. Detector functions are not scored in isolation, because a detector that returns a finding no deployment can see is not a detection — a distinction that cost us a harness to learn, and produced the cross-agent end-to-end reachability scorer.
| Term | Definition used throughout |
|---|---|
| Attack sample | One prompt, tool call, file or piece of retrieved content constructed to make an agent act against its user. |
| Benign sample | Legitimate developer or security work that must not be flagged. A hard negative is a benign sample deliberately shaped to resemble an attack. |
| Detection | A categorized finding at or above the alert threshold. |
| Prevention | An action the agent was stopped from executing. Only a sample that carries an action can be prevented. |
| Right reason | A catch credited to a detector that matches the sample's actual attack class, rather than a coincidental match by an unrelated rule. |
| False positive | A benign sample producing any finding at or above the alert threshold — regardless of category, because a wrong-category alert is still an alert the user has to deal with. |
| Enforcing policy | A device enrolled to a tenant whose policy sets the relevant category to block, as distinct from alert. |
Three stages of the product are exercised end to end: the tool stage (a tool call before it executes), the index stage (content as it enters the agent's context) and the output stage (what the agent produces). Which stage a sample can be reached at is a property of the sample, not a choice — supply-chain samples are tool metadata and config files reachable only at the tool, file and index stages, and cross-agent samples may be event-graph shapes with no text at all.
The outcome vocabulary
AMTSO asks for a distribution of outcomes rather than a single pass or fail, and these are the buckets a scored sample lands in. Two of them are not creditable to the product at all, which is the entire reason the vocabulary is worth having.
| Outcome | What it means, and who gets the credit |
|---|---|
| Prevented | The action was stopped before it executed. Requires an enforcing policy and a sample that carries an action; jailbreak and injection material has nothing to stop, so for that material the correct outcome is detection, not prevention. |
| Detected, not prevented | Flagged at or above the alert threshold, but nothing was stopped — either because the sample carries no action, or because policy is set to alert rather than block. A real result, and a weaker claim than prevention. The two are never merged into one blended "block rate". |
| Model refusal | The model declined unaided, with the product out of the path. Not creditable to the product — AMTSO states this explicitly, and it is the rule that the refusal baseline in section 07 exists to operationalise. |
| Model recognition | The model identified the attempt as an attack without the product in the path, whether or not it went on to refuse. Like refusal, this belongs to the model and is reported separately rather than folded into product coverage. |
| Missed | No finding at or above the alert threshold, and no refusal. The honest bucket, and the one whose contents are worth reading. |
| Inconclusive | The run does not adjudicate — the sample could not be shown to work with the product absent, the harness errored, or repeated runs disagree. Counted as its own outcome and never quietly redistributed into either success or failure. |
A local refinement sits on top of this: a catch is additionally scored as right-reason or not, and the right-reason rate is reported separately from recall wherever it is materially below it. An incidental catch — the sample flagged by a detector aimed at something else — is a real detection, but the categorization a security team sees is wrong, and it is a much weaker guarantee of generalisation, because it does not survive the attacker changing the one thing the unrelated rule happened to match. The full distribution of outcomes for a specific release, including the right-reason breakdown, is published in the benchmark report.
Eleven harnesses, every test seen failing first
Eleven measurement scripts in the agent repository. Each one is described by what it measures, because a harness that cannot state what would falsify it is not a measurement. All of them import the shipped detection engine rather than re-implementing it, and none of them mutates a detector or a corpus.
| Harness | What it measures |
|---|---|
| Red-team coverage evalscripts/redteam-eval.mjs | Detection coverage across the ten HackAgent families, deterministically and LLM-free: every sample carries a ground-truth shouldDetect label, so coverage, precision and recall are computed by comparing the engine's verdict to that label — no model, no judge, reproducible on any machine. An optional flag routes samples through the on-device model layer instead; the default run never imports it, so the headline number is never model-assisted. Current release: 101/102 deterministic coverage, and 100% (29/29) on this harness's held-out slice at 94% precision (4 false positives / 178). |
| Locked tune/test splitterscripts/split-heldout-v2.mjs | Splits a corpus once into a tune half and a locked test half, stratified by family and transformation axis so both halves carry the same mix — an unstratified split could hand one half every homoglyph sample and make the result meaningless. Within each bucket samples are sorted by id and assigned alternately, so the split is reproducible from the corpus alone with no stored seed. Tuning work may see only the tune half. |
| Adversarial mutation generatorscripts/mutate-attacks.mjs | Generates a fresh attack set by transforming each family's published technique along explicit axes — synonym and voice swaps, directive-in-code-comment / JSON / blockquote indirection, homoglyphs, letter-spacing, Caesar and ROT13, persona and persuasion recombinations. It deliberately does not read the detectors: steering mutations toward what the engine already catches would burn the set. Every transform except a bounded single-character typo axis is information-preserving and reversible, so the malicious core is recoverable and the sample is provably still an attack. |
| Closed-loop adversaryscripts/closed-loop-mutate.mjs | Models an adversary who learns: round 0 is the plaintext cores, round 1 every single-step transformation, and round N+1 every one-step extension of only the chains that evaded in round N. A one-shot generator understates risk because it never composes two evasions; this spends its budget only on productive directions and reports which transformation chains survive. Validity is mechanical, not asserted — every candidate is rendered backwards and must reproduce the malicious core character for character, or it is dropped before it is ever scored. |
| Benign false-positive scorerscripts/score-benign-v2.mjs | The other half of the question recall never answers: across 610 realistic benign developer and agent prompts, 269 of them adversarially shaped hard negatives, how often does the engine fire when it should stay silent? Current release: 3.32% (20/602), specificity 96.68%. Reports the false-positive rate per bucket and per hard-negative twin family, and for every false positive the detector and threat ids that fired. It reuses the same reducer as the recall benchmarks, so the two numbers are apples-to-apples. |
| Model-refusal baselinescripts/measure-refusal-baseline.mjs | Per attack family, how often the model refuses on its own with no product involved — run N times because a single run is a point estimate. This is what converts a raw coverage figure into marginal value: recall added where the model already refuses is worth close to nothing. Its most useful output has been negative — it showed that an entire detection wave we had shipped bought almost no marginal protection, because the model refused those families anyway. |
| Variance harnessscripts/measure-variance.mjs | Repeated runs of the same scored set, reporting the distribution and per-sample stability rather than one number. It exists because we watched model-assisted figures move between runs — a judge recovering eight attacks once and five the next time, a benign false-positive count drifting by one. AMTSO asks for the distribution of outcomes rather than hiding variability behind a single pass or fail; this produces it. Measurement only: it mutates no detector, no corpus and no other script. |
| Action-layer blocking validatorscripts/moorai-validate-blocking.mjs | The one that tests prevention rather than detection. It assumes the agent has already lost — hijacked, jailbroken, whatever — and asks whether the malicious tool call it then attempts is denied. For each action it feeds a real PreToolUse payload to the real hook as a subprocess and records the actual decision from its stdout; it does not re-implement the hook's logic, which is the entire point. Current release: 12 of 12 malicious tool calls stopped under an enforcing policy. This is the harness behind any claim that uses the word "blocked". |
| Per-vector scorersscripts/score-vectors.mjs | Scores the two AMTSO attack vectors that ship detectors but had never been measured: tool / skill / extension / MCP supply chain and memory, context and cross-agent propagation. They need their own scorer because neither fits "one sample = one string": supply-chain samples are tool metadata and config files only reachable at the tool, file and index stages, and cross-agent samples are sequences — written in one session, consumed in a later one — or event-graph shapes with no text at all. Optimistic and strict readings are reported separately and are not interchangeable. |
| Cross-agent end-to-end reachabilityscripts/score-vector5-production.mjs | Whether a real deployment can actually see the cross-agent detections, as opposed to whether the library function returns a finding. It refuses to call the library at all: it seeds the on-device event window and then drives the two paths a deployment really has — the real PreToolUse hook, and the real agentwatch emitter — counting only what arrives at a stand-in management server. It was written because the library scored full marks on a capability nothing on the enforcement path was calling. |
| AMTSO dimension-coverage reporterscripts/amtso-coverage.mjs | Classifies every corpus sample along AMTSO's six dimensions and reports the distribution, flagging zero-coverage cells by name rather than reporting an average. Labels live in a sidecar; the corpora are never modified. AMTSO's own values are used verbatim and any local extension is prefixed and reported separately so it can never be mistaken for AMTSO coverage. Its job is to find the holes in our testing, and it does — see limits. |
Five rules, each of which cost us a number
Anyone can write a scoring script. The five rules below are what make its output worth reading, and each of them has cost us a number we would have preferred to publish.
01 Locked held-out splits the tuning work never sees
A corpus is split once, stratified, and the test half is off-limits to detector work — not "we tried not to look at it", but a separate file that the tuning wave does not open. We enforce this because we broke it once: an earlier held-out set was tuned against and thereby destroyed as a generalisation measure, and it has been retired rather than quietly re-used. The gap between the tune half's score and the locked half's score is published as the overfit margin, because that gap is the finding.
02 Baseline validation — a sample only counts if the attack works with the product off
Blocking something that was never going to succeed is not protection. Two baselines run underneath the coverage numbers: the refusal baseline, which removes credit for anything the model would have refused anyway, and the enforcement baseline, which confirms an action actually executes when the product is not in the path. AMTSO states the refusal principle; the practical consequence is a priority rule we now follow — measure the refusal baseline for a family first, and build only where the model complies.
03 False positives measured alongside, reported separately
Every recall figure has a precision figure measured on the same engine against the 610-prompt benign corpus, and the two are never combined into a single flattering score. The corpus is deliberately hostile to us: 269 of its 610 samples are hard negatives — legitimate security questions, benign role-play, urgent-sounding ordinary requests, exfiltration-shaped but innocent prompts, and 101 deliberately obfuscated benign samples (base64 blobs, minified JavaScript, non-English prose, leetspeak identifiers, ASCII tables) added because the previous corpus contained exactly one — grouped into twin families so we can see which kind of benign prompt we misfire on, not just how often.
04 Falsify first — every test seen failing before it is trusted
A test that has never been observed failing is not evidence; it is a green light with no wiring behind it. So the production path is deliberately broken, the test is confirmed to go red, the break is reverted, and what was broken is recorded. This is the rule that catches the embarrassing class of bug — the harness that scores a capability nothing in the shipping product calls, which is exactly what the cross-agent end-to-end reachability harness was written to expose after a library-level score of full marks turned out to be unreachable in a real deployment.
The clearest example of why mutation is the only version of this that works is a benign fixture that produced zero findings at every stage. It carried an assertion that its clean result must not silently mean "never scanned" — and that assertion was green. It would also have been green on the day the scan was deleted, because from outside the harness zero findings and no scan are the same observation. Only breaking the production line — removing the scan and confirming the test goes red — distinguishes them, and until that is done the assertion is decoration. Every claim on this page has a test that has been seen red.
05 Repeated runs, variance reported instead of a lucky pass
Anything involving a model is a sample from a distribution. A single passing run is a point estimate presented as a fact, and we have watched those estimates move — which is why model-assisted figures are run N times and published as a spread, and why the deterministic, LLM-free number is always the headline while the model-assisted one is reported separately and never merged into it.
Reproducible on any machine, against a pinned model digest
"Re-run it and disagree" is the only offer we can make in place of an independent lab, so it has to be a real one. Four properties are what make it real.
01 The headline path is deterministic and LLM-free
Every sample carries a ground-truth shouldDetect label, and the default scoring run compares the engine's verdict to that label with no model anywhere in the loop. Coverage, precision and recall from that path are byte-identical on any machine. The model-assisted path exists behind an explicit flag; the default run never imports it, so a headline number is never quietly model-assisted.
02 Splits are fixed and derivable, not seeded
The tune/test split is recomputed from the corpus by sorting and alternating rather than sampling, so there is no seed to publish and no seed to get wrong. Corpus generation is seeded and fixed. Reproducing our halves requires the repository and nothing else — see section 05.
03 The model is pinned by digest
Where a model is involved at all — the refusal baseline and the semantic escalation judge — it is one pinned local build: llama3:latest, 8.0B parameters, Q4_0, digest 365c0bd3c000. A tag alone would not be reproducible, because a tag moves; the digest is what makes "the same model" checkable a year from now.
04 Everything runs on-device, so anyone can run it
There is no API key, no scoring service and no network call anywhere in the measurement path. Attack corpora are never sent to a third party, results are never adjudicated by a remote judge, and the cost of a run is electricity — which is what makes repeated runs, and therefore honest variance reporting, affordable at all. Model-assisted figures are run N times and published as a spread, never as a lucky single pass and never merged into the deterministic number. The agent, the harnesses and the corpora are MIT.
Scope, stated once, because a figure without one is not citable:
| Scope item | What is covered |
|---|---|
| Architecture | On-device. Detection runs locally; a category, a risk level and a one-way hash are the only things that leave the machine. Content-free by construction, which is also why no corpus can be built from user telemetry. |
| Stages measured | Tool stage, index stage and output stage, each exercised through the shipping product end to end rather than at the library boundary. |
| Hosts covered | Host-level hooks, plus MCP enforcement that is host-independent — it applies wherever an agent reaches a tool over MCP. Not every agent host is registered; the release-specific list is in the benchmark report. |
| Enforcement scope | Prevention claims apply to a device enrolled to a tenant with an enforcing policy. An unenrolled device enforces no tenant policy and blocks no outbound action — by design, not as a gap. |
| Corpora | A family-keyed HackAgent set, a mutation-generated set split into tune and locked halves, a locked held-out set of 44 attacks and 10 benign samples, a 610-sample benign corpus with 269 hard negatives, and four per-vector corpora. |
| Independent review | None. Every figure is self-measured, on self-authored corpora, on our own hardware. |
What these numbers do not establish
Stated plainly, because a limit you have to discover for yourself is a limit that was hidden. These are the ones that would most change how a result should be read.
Every attack sample is written by us. None is a captured live incident, and no telemetry from a real deployment feeds the corpora — a direct consequence of the product being content-free, and therefore a limitation we cannot engineer away by collecting more data. What the results describe is how the product behaves against attacks we know how to write. The mutation and closed-loop harnesses exist to widen that set beyond our own imagination, and the refusal baseline exists to stop us crediting ourselves for the easy half of it, but neither converts a constructed corpus into field evidence.
Prevention — the action-layer blocking numbers — is measured against a single agent host, because that is where a PreToolUse hook exists to enforce at. Other hosts are covered at the detection layer, and a detection claim is not an enforcement claim. Any statement that a malicious tool call was blocked should be read as scoped to the host it was measured on.
The refusal baseline and the semantic judge both run on llama3:latest (8.0B, Q4_0). A frontier model refuses more, and decodes obfuscated payloads far better. Inspecting our own runs showed the 8B model was frequently not complying with obfuscated attacks so much as failing to decode them — it read base64 and hallucinated the plaintext, misread leetspeak as a cipher, botched reversals. A frontier model would decode those correctly and might then refuse, which means a meaningful share of the marginal value we measure for the obfuscation families is soft, and we say so rather than banking it.
Labelling every classified attack sample against the AMTSO taxonomy gives 624 samples, of which 296 (47.4%) are vector 1, direct input — 130 are vector 2 (indirect content), 78 vector 3 (tool / MCP supply chain), 62 vector 4 (outbound action), 42 vector 5 (memory / cross-agent propagation), and 16 are static code artifacts carrying no vector. All four of the non-direct vectors now have dedicated, sub-technique-enumerated corpora and their own scorers, so no vector is starved; vector 1 is still the largest single share by a wide margin, because it remains the easiest to author. The reporter prints the distribution every run.
The sharper limit is inside vector 2. The dedicated set scores 73.3% recall at 78.6% precision, but only 11 of the 33 catches fire for the right reason — the rest are caught incidentally by a detector aimed at something else, and an incidental catch does not survive the attacker changing one thing. data-poisoning-fact, where poisoned content changes the agent's answer while issuing no directive at all, sits at 25%. That is a rule gap rather than a structural one. The delivery path underneath it — inbound WebFetch and WebSearch content, the channel indirect prompt injection actually arrives on — is scanned at the output stage, where 22 of 24 output-stage attack samples are caught. A PreToolUse hook cannot read that content, because the page does not exist yet when it fires, so the scan runs on the result instead. The consequence is directional: on inbound content the product detects and warns but does not prevent, since the fetch has already happened by the time the result is visible. And the 73.3% above is unmoved by any of it — that figure is measured against the detectors directly rather than through the shipped hook, so it is a statement about right-reason attribution, not about reach.
Every figure here is self-run, on self-authored corpora, on our own hardware. No third-party lab has verified, reproduced or audited any of it. The mitigation we can offer is not a certificate but a repository: the agent, the harnesses and the corpora are open source under MIT, so the falsifiable version of this page is somebody re-running it and disagreeing.
We read AMTSO's published Guidelines for Testing of Agentic Security Products v1.0 and graded our own testing against its criteria. That is self-assessment, not accreditation. AMTSO has not reviewed our methodology, has not tested our product, has not certified anything, and has no relationship with us. Nothing on this page is an AMTSO result, an AMTSO score, or an AMTSO endorsement, and any reading of it as one is wrong. The same applies to OWASP, MITRE and every other body named here.
Run your own tool through this, and grade us the same way
The method above is the whole method. Nothing in it is held back for a sales call, and none of it takes our word for anything.
The agent, the eleven harnesses and every corpus are in the open-source repository under MIT. The split recomputes from the corpus with no stored seed, the headline path is deterministic and LLM-free, and the model is pinned by digest — so a run on your machine and a run on ours are the same run, and a disagreement about a result is checkable rather than a matter of trust. Point the same harnesses at your own guard: label the samples along the same six dimensions, validate that each attack works with the product removed, subtract your model's refusal baseline, and publish the distribution rather than a headline. Grade us by the same standard, and tell us where a number here does not survive it.
The method and the headline results are both on this page, ungated. The full breakdown for a specific release — the outcome distribution, per-family and per-vector coverage, per-device-state enforcement, the false-positive breakdown by twin family and the overfit margin — is in the full benchmark report.
This page describes the method and is deliberately ungated; the headline figures are for the current release and the full breakdown behind them is published in the full benchmark report. Harness paths refer to the open-source MoorAI agent repository (MIT). Corpus sizes and the AMTSO dimension distribution above were produced by running the in-repo reporter; they move as corpora grow, so treat them as of the current release rather than as fixed constants. AMTSO has not reviewed, certified or endorsed MoorAI, and the Guidelines are cited as published criteria we graded ourselves against. OWASP® is a trademark of the OWASP Foundation; ATT&CK® and ATLAS™ are trademarks of The MITRE Corporation; AMTSO is a trademark of the Anti-Malware Testing Standards Organization. Llama 3 is used under the Meta Llama 3 Community License. No third-party penetration-testing or LLM red-teaming tool was used to produce any figure referenced on this page.