The Agentic Security Benchmark

Recall, precision, F1, over-blocking and latency, each with the set it was measured on

Security vendors often publish F1, recall, false-positive and latency figures for their detection models. These are MoorAI's. Each rate below is computed when the site is built, from raw counts copied from the scorers' output, and each row names its set and size. Precision, recall and F1 are never computed across two different sets.

Scroll sideways →

Where these come from. Everything except the public-benchmark row is measured in the MoorAI agent repository at commit 4f76f18 (v0.96.0), on 2026-09-27, from a clean export of that commit. The raw counts are vendored into this site with that provenance and every rate is derived from them at build time. The model verdicts behind the marginal rows were recorded on 2026-09-06 (Llama 3 8B) and 2026-09-07 (Claude Sonnet) and are re-scored against today's detectors. The refusal script's own printout credits MoorAI with one more catch against Claude Sonnet, because it scans a multi-turn sample as one joined text. The scorer behind the headline recall scans it turn by turn and misses it, so this table counts it as a miss everywhere.

Who wrote the samples. The benign corpus was hand-written by MoorAI's author, who also wrote the detectors. The attack set was generated by a mutation generator from seeds the same author wrote, across the ten HackAgent families. None of it is captured from real incidents, and no third party has reproduced these figures.

Latency. Two runs on an Apple M5 Max, 18 cores, 48 GB, macOS 27.0, Node v22.22.0, with a load average 13 to 15 during both runs: the machine was running other work; the table shows the slower run at each percentile. The engine figure is decideText at the prompt stage, warm, 5 passes over the 610 benign samples. The hook figures time one node cli/moorai-hook.mjs process per event on an unenrolled device with a sandboxed home directory: one prompt per benign sample, and 200 Bash calls with ordinary commands. The timing harness is a short script, not in the repository; the method above is all it does.

A detection rate silently includes attacks the model would have refused anyway

Published detection rates for AI-agent security products are mostly uninterpretable. A percentage with no stated corpus, no false-positive line, no held-out split and no model-refusal baseline is a number about nothing, and two products quoting the same figure can be adding wildly different amounts of protection.

The full argument is in the repository's AMTSO.md: the outcome vocabulary, the six classification dimensions and baseline validation. How the same discipline is applied to MoorAI's own numbers is on How we test.

Seven corpora, every sample labelled

Each file is self-describing: a header comment says why the corpus exists and how it was built, and most attack samples carry a per-sample validity statement and an AMTSO label object. The counts below are read from the benchmark's own scored output, not typed by hand.

Scroll sideways →

Of the 286 attack samples, 225 carry a written validity statement and 216 carry AMTSO labels; 102 carry all four label dimensions. The gaps are per corpus and are itemised in the repository's corpora README. Two structural gaps are listed there: vector 1 (direct input) is over-represented across the wider labelled set because it is the easiest vector to author, and vector 6 (static code artifact) has no dedicated corpus.

A sample counts only if the attack could succeed with the product absent

This is AMTSO's rule, and it keeps a corpus from being padded. An attack that resolves to nothing executable, that targets a capability the agent does not have, or that would fail for its own reasons is an invalid sample: neither a miss nor a catch.

Part of the corpus is missing on purpose

Two held-out splits are left out of the benchmark corpora: the test half of the web-content benign corpus (153 samples) and a separate held-out attack set. Only the tune halves ship here. The locked halves are not secret: they are in the public MoorAI repository, where our detector work never opens them. A product that has seen them can tune against them, so treat a score on them as a check for products scored before that, not as proof.

One required method

An adapter is a single ES module. Only scanText is required. Optional scanAction, scanSession and scanEvents enable the action, sequence and event-graph samples, and you are not penalised for omitting them: samples whose harness you do not implement land in a not-applicable bucket and are excluded from every rate, so a text-only product does not appear to have missed samples it was never shown.

# clone, then run the naive reference adapter to see the output shape
git clone https://github.com/gitayg/agentic-security-benchmark
cd agentic-security-benchmark
npm run score -- --adapter keyword

# then score your own
node scorers/run.mjs --adapter ./acme-adapter.mjs --corpus all --json > results/acme.json

Corpora only get better if people who are not the maintainer add to them. There are structured issue forms for proposing an attack sample, proposing a benign sample, reporting a mislabelled sample, and submitting results for another product; see CONTRIBUTING.md. Additions to a held-out split are not accepted from outside, because that would compromise the split.

3 scored runs, and none of them independent

The results directory holds one file per product and is built to hold more than one. Every run currently in it was executed by the benchmark maintainer, who is also the author of MoorAI; the MoorAI result says so in its own conflictOfInterest field. These are not independent tests. No third-party lab has reproduced anything here.

Two of them exist to make the third readable: a null adapter that detects nothing, so the floor of every rate is visible, and a naive keyword adapter, so a score can be compared against something rather than against nothing.

Every scored run, with its per-vector breakdown, is on the leaderboard. A vendor who wants a row opens a pull request against the benchmark repository.

Run the proof of concept on samples the vendor did not pick

A vendor demo shows the prompts the vendor knows it catches. These steps work for a proof of concept with MoorAI or with anyone else, and each one removes a way a detection claim can look better than the protection behind it.

1. Score the product on a public corpus. Clone the benchmark, write an adapter around the vendor's product (only scanText is required), and run node scorers/run.mjs --adapter ./vendor-adapter.mjs --corpus all --json. Every product in the evaluation then faces the same samples.
2. Find out what the model refuses before crediting the product. Send each attack to the bare model with no product in the path, several times per attack family, because one run is a single draw. Credit the product only for catches among the attacks the model lets through. That remainder is its marginal detection. The harness is model-free and reports model refusal as zero, so this step is yours to run.
3. Check whether the tool call ran. A flagged prompt is not a stopped action. Drive the product's real enforcement path through scanAction and record whether each call executed. Only justify, block or kill count as prevention, and every vector 4 row landing in not-applicable means the action surface was never tested. Note the policy posture you ran at, and confirm each attack works with the product removed (section 03). For MoorAI, run this step on Claude Code, the host where enforcement is validated: the Codex and Copilot CLI adapters block through those agents' pre-tool hooks but are not yet validated end to end against the live agents.
4. Count false positives in the same run. The benign corpora are scored alongside the attacks, including 269 hard negatives in benign-corpus-v2. A catch rate with no false-positive rate from the same engine and settings tells you nothing about what your developers will put up with.
5. Keep a held-out set the vendor never sees. Split your own samples with the published algorithm (section 04), share nothing from the test half, and score it once, at the end. A vendor that tunes on your samples will score well on them. Our own locked halves sit outside the benchmark corpora but are readable in the public MoorAI repository, so they cannot play that role for a vendor that has read them.

Read MoorAI's published numbers the same way: they are raw detection rates with no refusal baseline subtracted, the 61/61 is a tune-half score, and vector 4 prevention is 18 of 57 at the default posture and 43 of 57 under an enforcing policy.

Not certified, and self-graded

Coverage maps, reports and research

Where each mapping and measurement is published in full.

• MITRE ATT&CK v20 coverage: every rule’s credits, technique by technique
• MITRE ATLAS coverage: the agentic techniques, vendor by vendor
• ATLAS: model providers: the providers’ own safety docs against the same techniques
• Synthetic Insider Threat Matrix: the matrix read against MoorAI
• Attack families: what a product adds over model refusal, family by family
• Threat rule-base: every rule, mapped to OWASP, ATLAS, ATT&CK and STRIDE
• Hidden Unicode scan: invisible characters in what AI agents read
• AMTSO guidelines grading: MoorAI graded against AMTSO’s agentic testing guidelines
• Leaderboard: every scored run, read from the benchmark repository
• AMTSO benchmark report (PDF): the full outcome distribution for a release
• AI risk guide (PDF): each AI risk and how to mitigate it