Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Agent Evals with Real Statistics

    1

    Know whether an agent or prompt change is actually better: golden datasets, calibrated LLM judges, confidence intervals, paired comparisons, power analysis and CI regression gates.

    $12

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Agent Evals with Real Statistics

    Agent Evals with Real Statistics

    Example session with this skill installed

    We changed our support agent's system prompt. On our 40-question test set the old prompt passed 29 and the new one passed 33, one run each. My manager wants to ship it because it's '10 points better'. Is that conclusion justified, and what eval setup should we use going forward? Answer in eval_plan.md.

    • Read your context and instructions
    • Compiled the agent evals

    eval_plan.md: Not justified yet. 29/40 = 72.5% (95% Wilson CI 57–84%) and 33/40 = 82.5% (CI 68–91%) overlap heavily, and with 40 cases × 1 run the smallest difference a paired test can reliably detect is about 18 points. Plan: run both prompts on the same cases with 3 reps each, compare per-case means with a paired bootstrap and McNemar; to detect a 10-point gain at 3 reps you need about 45 cases; separate timeouts and rate limits from model failures; add the 11 currently failing cases to a regression gate in CI.

    eval_plan.md

    Connects securely to your tools. The creator never sees your data.

    About this skill

    An eval is a measuring instrument. Before trusting its numbers, make sure it measures the right thing, separates the model from the plumbing, and has enough resolution to see the change you care about. Most "surprising" eval results are eval bugs.

    What your agent does with it

    • Case design from reality: production transcripts and incidents, cases where the agent should and should not act, unambiguous pass criteria, ground-truth provenance, no leakage, tags for slicing.
    • Graders that measure outcomes: exact/normalised match, programmatic checks, end-state checks for agents that act, rubric and pairwise LLM judges, human review for calibration. Atomic properties instead of one blended score.
    • LLM-as-judge done properly: judge prompt template, a calibration protocol (≥90% agreement with humans, known-negative tests, flip rate), cost control.
    • A harness that does not lie: infrastructure errors kept out of scores, fresh sandbox per trial, production config, full transcripts, multiple reps, served-model check. Resumable runner skeleton and results schema included.
    • Statistics that prevent false wins (eval_stats.py): Wilson and bootstrap intervals, paired bootstrap comparison with McNemar test, power / minimum detectable effect, required cases, pass@k and pass^k.
    • Regression gates in CI: fast smoke suite on every change, full suite nightly, gates on capability tags, safety cases and cost.

    Included: SKILL.md, harness and LLM-judge references, eval_stats.py (standard library only), and an eval case.

    Part of the Production AI Agents series. Start free with Agent Architecture; pair with Agent Guardrails (red-team cases) and Text-to-SQL Data Agent (execution-accuracy evals).

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Does not cover operational telemetry such as latency, error and cost dashboards (see agent-observability) or launch sign-off (see agent-production-readiness). It does not run your agent for you; it designs the suite, harness and analysis.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 1 day ago

    Software
    Python 3.10+

    Permissions

    File Scopes

    Your
    project
    folder
    (writes
    case
    sets
    grader
    code
    harness
    and
    reports)

    eval_stats.py runs locally with the Python standard library and makes no network calls.

    Listed1 day ago

    What's inside

    Frequently Asked Questions

    No. The statistics script uses only the Python standard library and reads plain JSONL results. It works alongside any eval platform you already use.

    It tells you whether a difference is real: confidence intervals, paired comparisons on the same cases, power analysis for how many cases you need, and separation of infrastructure errors from model failures.

    Yes: a judge prompt template, a calibration protocol against human labels, known-negative tests and cost control.

    New in LLM Ops