Works with the AI Agents you already use
Agent Evals with Real Statistics
Know whether an agent or prompt change is actually better: golden datasets, calibrated LLM judges, confidence intervals, paired comparisons, power analysis and CI regression gates.
$12
Agent Evals with Real Statistics
Example session with this skill installed
We changed our support agent's system prompt. On our 40-question test set the old prompt passed 29 and the new one passed 33, one run each. My manager wants to ship it because it's '10 points better'. Is that conclusion justified, and what eval setup should we use going forward? Answer in eval_plan.md.
- Read your context and instructions
- Compiled the agent evals
eval_plan.md: Not justified yet. 29/40 = 72.5% (95% Wilson CI 57–84%) and 33/40 = 82.5% (CI 68–91%) overlap heavily, and with 40 cases × 1 run the smallest difference a paired test can reliably detect is about 18 points. Plan: run both prompts on the same cases with 3 reps each, compare per-case means with a paired bootstrap and McNemar; to detect a 10-point gain at 3 reps you need about 45 cases; separate timeouts and rate limits from model failures; add the 11 currently failing cases to a regression gate in CI.
Connects securely to your tools. The creator never sees your data.
About this skill
An eval is a measuring instrument. Before trusting its numbers, make sure it measures the right thing, separates the model from the plumbing, and has enough resolution to see the change you care about. Most "surprising" eval results are eval bugs.
What your agent does with it
- Case design from reality: production transcripts and incidents, cases where the agent should and should not act, unambiguous pass criteria, ground-truth provenance, no leakage, tags for slicing.
- Graders that measure outcomes: exact/normalised match, programmatic checks, end-state checks for agents that act, rubric and pairwise LLM judges, human review for calibration. Atomic properties instead of one blended score.
- LLM-as-judge done properly: judge prompt template, a calibration protocol (≥90% agreement with humans, known-negative tests, flip rate), cost control.
- A harness that does not lie: infrastructure errors kept out of scores, fresh sandbox per trial, production config, full transcripts, multiple reps, served-model check. Resumable runner skeleton and results schema included.
- Statistics that prevent false wins (
eval_stats.py): Wilson and bootstrap intervals, paired bootstrap comparison with McNemar test, power / minimum detectable effect, required cases, pass@k and pass^k. - Regression gates in CI: fast smoke suite on every change, full suite nightly, gates on capability tags, safety cases and cost.
Included: SKILL.md, harness and LLM-judge references, eval_stats.py (standard library only), and an eval case.
Part of the Production AI Agents series. Start free with Agent Architecture; pair with Agent Guardrails (red-team cases) and Text-to-SQL Data Agent (execution-accuracy evals).
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
Does not cover operational telemetry such as latency, error and cost dashboards (see agent-observability) or launch sign-off (see agent-production-readiness). It does not run your agent for you; it designs the suite, harness and analysis.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 1 day ago
- Software
Python 3.10+
Permissions
File Scopes
eval_stats.py runs locally with the Python standard library and makes no network calls.