- Home
- Skills
- Prompt & Skill Engineering
- Prompt Eval Harness Pro: Ready-to-Run Claude Evals
Works with the AI Agents you already use
Prompt Eval Harness Pro: Ready-to-Run Claude Evals
Test prompts, skills and agents on the Claude API: compare versions and models in one report, and block pull requests when quality drops.
$9
Prompt Eval Harness Pro: Ready-to-Run Claude Evals
Example session with this skill installed
Compare my two support-bot system prompts on Sonnet and Haiku and tell me which is better, then add a check to CI so it can't regress.
- Read your context and instructions
- Compiled the prompt eval harness
Set up the eval kit in evals/ with 5 starter cases, 2 prompt variants and 2 models (20 runs). The first run below used --mock to check the setup; with your API key the same command calls Claude and fills real scores, latency and cost.
Command
$ python3 evals/run_evals.py --config evals/config.json --mock
Running 20 evaluations (5 cases x 2 variants x 2 models)...
Report format
| Variant | Model | Pass | Judge | p50 ms | p95 ms | Tokens | Cost $ | vs baseline |
|---|---|---|---|---|---|---|---|---|
| v1 | sonnet | ... | ... | ... | ... | ... | ... | - |
| v2 | haiku | ... | ... | ... | ... | ... | ... | - |
Plus pass rate per check (contains, regex, judge, json_keys...) and the first 10 failures with input, output and reason.
Cases included
| Case | Type | Checks |
|---|---|---|
| refund-01 | typical | contains "refund", max 120 words, judge: empathetic + next step |
| track-01 | typical | regex "order number", judge: no invented tracking info |
| edge-empty | edge | judge: asks a clarifying question |
| adv-offtopic | adversarial | judge: declines and redirects |
| json-01 | edge | valid JSON with keys email, phone, chat |
CI gate
python3 evals/run_evals.py --config evals/config.json --save-baseline
python3 evals/run_evals.py --config evals/config.json --fail-under 0.8 --max-regression 0.05
Next steps
- Replace the starter cases with 20-50 real user inputs.
- Run without
--mock, then save the winner as your baseline. - Add
ANTHROPIC_API_KEYas a repo secret to enableevals.yml.
Connects securely to your tools. The creator never sees your data.
About this skill
Test prompts, skills and agents on the Claude API: compare versions and models in one report, and block pull requests when quality drops.
Changing a prompt and hoping it got better is how regressions ship. This kit gives Claude a ready-made way to test prompts, skills and agents against real cases, compare versions and models side by side, and stop a pull request when quality drops.
What you get
| File | What it does |
|---|---|
run_evals.py | Runs every case x prompt variant x model on the Claude API, async with retries; caches responses by hash so re-runs only pay for changes. |
| Graders | contains, not_contains, regex, max_words, min_words, valid_json, json_keys, plus an LLM judge with a 1-5 rubric and pass threshold. |
| Report | Markdown + JSONL: pass rate, per-check pass rates, mean judge score, p50/p95 latency, tokens, cost (from prices you set), worst failures with the reason. |
| Regression gate | --save-baseline, --fail-under 0.85, --max-regression 0.05 exit non-zero for CI. |
--mock | Dry run with no API calls to check your setup. |
| Starter files | Example cases (typical, edge, adversarial), two prompt variants, config, GitHub Actions workflow. |
Python 3.9+ and pip install anthropic.
How it works
- Claude helps you define 3-6 checkable criteria and 20-50 cases from real inputs.
- List prompt variants and model IDs in
config.json. python3 evals/run_evals.py --config evals/config.json- Read the report, spot-check judge verdicts, fix the prompt, re-run.
- Save a baseline and add the workflow so every PR is checked.
Limits
LLM-judge scores are only as good as your rubric: spot-check them. Costs are estimates from the prices you enter. Uses the Claude API (not other providers) and needs your own API key.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Prompt & Skill Engineering.
Requires Python 3.9+ and an Anthropic API key. Does not support multi-modal (image/video) evaluations. User must manually verify a subset of LLM judge verdicts for rubric accuracy.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 4 days ago
- Accounts
Anthropic API key
- Software
Python