Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Detection Rule Benchmark

    1

    Measures your security detection rules against labelled attacks and benign lookalikes, then reports detection rate, false positives, error bars, and missed cases, with a pass or fail gate for CI.

    $12

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Detection Rule Benchmark

    Detection Rule Benchmark

    Example session with this skill installed

    Here are our test scenarios and the alerts from last night's rule run. How good are these detections really, and what should we fix first?

    • Read your context and instructions
    • Compiled the detection rule benchmark

    This uses examples/, which ships with the skill, with --known-rules listing a fourth rule. The numbers are real output from running score_detections.py on that sample data, so you can reproduce them.

    Summary

    Status: fail. Detection is 5 of 6 and false positives are 1 of 5, so both targets are missed. The set is also too small to support a pass.

    Results

    MetricResult95% intervalTarget
    Detection rate83.3% (5 of 6)43.7% to 97.0%at least 90%
    False positive rate20.0% (1 of 5)3.6% to 62.5%at most 10%
    Hard negative false alarms33.3% (1 of 3)not gated
    Technique recall62.5%not gated
    Technique precision100%not gated

    These intervals are wide because the sample is small. Treat the rates as rough.

    What to fix first

    DNS tunnelling is never detected. dns-tunnel-01 was missed, and the dns-long-labels rule never fired, so check that rule and its data source first.
    2.

    powershell-bypass fires on a developer build. Scenario dev-build-01 is a false alarm. It needs to separate build tools from attacker behaviour.
    3.

    The PowerShell rule doesn't report obfuscation. Both encoded-ps scenarios expect T1027.010, but the alerts only report T1059.001, which is part of why technique recall is 62.5%.

    Per family

    FamilyDetected
    brute-force2 of 2
    script-execution2 of 2
    ransomware1 of 1
    dns-tunnel0 of 1

    What the data can't support

    The script also reports these shortfalls, so fixing the rules alone won't produce a pass:

    {
      "status": "fail",
      "problems": [
        "detection rate 83.3% is below 90.0%",
        "false positive rate 20.0% is above 10.0%",
        "family dns-tunnel was never detected"
      ],
      "shortfalls": [
        "6 attack scenarios, need 20",
        "5 benign scenarios, need 20",
        "3 hard negatives, need 5"
      ]
    }
    

    Next steps

    1. Fix the DNS data source and the PowerShell rule, using tuning scenarios only.
    2. Grow the set to at least 20 attacks, 20 benign scenarios, and 5 hard negatives, with a holdout.
    3. Save the new report as a baseline and add the script to CI.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    "100% detection" means little if the test was five easy attacks. A low false positive rate means little if the benign data looked nothing like an attack. This skill gives your agent a fairer way to test detection rules, and a script that reports the result with its uncertainty.

    What it does

    • Tests against hard negatives. Benign activity built to look like an attack, such as an admin creating accounts or a developer running PowerShell with -ExecutionPolicy Bypass, is tracked and scored separately.
    • Reports error bars. Every rate comes with a 95% Wilson interval, so 20 of 20 is reported as about 84% to 100%, not as 100%.
    • Won't pass on thin data. Fewer than 20 attacks, 20 benign scenarios, or 5 hard negatives returns insufficient, even with perfect scores.
    • Finds gaps that averages hide. An attack family that is never detected fails the run, even when the overall rate meets the target.
    • Keeps a holdout. Mark scenarios as holdout and the gate scores only those, which guards against tuning to the test.
    • Catches regressions. Compare with an earlier report, and any newly missed attack or new false alarm fails the run.
    • Names what to fix. Missed scenarios, the rule behind each false alarm, rules that never fired, and alerts that matched nothing.

    What's in the package

    • SKILL.md: the routine your agent follows
    • scripts/score_detections.py: the scorer, in Python standard library only, with no network calls
    • references/scenario-design.md: how to vary attacks, hard negatives to include, and how much data you need
    • references/file-formats.md: input formats, options, and report fields
    • examples/: a small scenario set you can run

    Good fit

    • Writing or tuning SIEM, Sigma, KQL, Splunk, YARA-L, or Elastic rules
    • A CI check on a detection-as-code repository
    • Giving leadership detection numbers that hold up

    Known limitations

    • It doesn't run your rules. You run them on your own engine and export the alerts. It works with any engine that can do that, but it has no connectors or rule converters.
    • It doesn't include attack data. You supply recorded or synthetic telemetry. The guide explains how to design it.
    • A pass applies to these scenarios only. It doesn't mean you would catch a real attacker.
    • It can't detect made-up alerts or labels. The skill tells the agent not to work out matches by reading rules, but the score is only as good as its input.

    This is a tool for defensive measurement. It contains no attack tooling.

    Where it comes from

    The approach comes from the evaluation of Aegis, my AI security operations platform, which is benchmarked against synthetic attacks and benign lookalikes. The results published for Aegis measure that system, not this skill.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Security.

    Requires Python 3.9+. Doesn't run your rules, convert formats, or include telemetry. A pass applies to the tested scenarios only. It can't detect made-up alerts or labels.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 15 days ago

    Software
    Python 3.9+
    SIEM
    orlocal rule runner

    Permissions

    Terminal / Shell
    Read Files
    Write Files

    Terminal runs scripts/score_detections.py. Read covers your scenario and alert files. Write covers saved reports. The script makes no network calls.

    Listed15 days ago

    What's inside

    Frequently Asked Questions

    No. You run them on your own engine and export the alerts as JSON. The scorer compares those alerts with your labelled scenarios, so it works with any engine that can export which events an alert covered.

    SKILL.md with the routine, score_detections.py (Python standard library, no network calls), a scenario design guide, a file format reference, and a small example set.

    No. It's for defensive measurement. You supply recorded or synthetic telemetry, and the guide explains how to design scenarios and hard negatives.

    Small tests can't support strong claims. 20 of 20 still has a 95% interval of about 84% to 100%. The defaults need 20 attacks, 20 benign scenarios, and 5 hard negatives, and you can change them with options.

    Yes. The exit codes are 0 for pass, 10 for fail, 11 for insufficient data, and 2 for bad input. With --baseline, any newly missed attack or new false alarm fails the run.

    No. It means your rules met your targets on these scenarios. The report should also say what the test doesn't cover, such as techniques with no scenario.

    New in Security