Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Prompt Eval Harness Pro: Ready-to-Run Claude Evals

    1

    Test prompts, skills and agents on the Claude API: compare versions and models in one report, and block pull requests when quality drops.

    $9

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Prompt Eval Harness Pro: Ready-to-Run Claude Evals

    Prompt Eval Harness Pro: Ready-to-Run Claude Evals

    Example session with this skill installed

    Compare my two support-bot system prompts on Sonnet and Haiku and tell me which is better, then add a check to CI so it can't regress.

    • Read your context and instructions
    • Compiled the prompt eval harness

    Set up the eval kit in evals/ with 5 starter cases, 2 prompt variants and 2 models (20 runs). The first run below used --mock to check the setup; with your API key the same command calls Claude and fills real scores, latency and cost.

    Command

    $ python3 evals/run_evals.py --config evals/config.json --mock
    Running 20 evaluations (5 cases x 2 variants x 2 models)...
    

    Report format

    VariantModelPassJudgep50 msp95 msTokensCost $vs baseline
    v1sonnet..................-
    v2haiku..................-

    Plus pass rate per check (contains, regex, judge, json_keys...) and the first 10 failures with input, output and reason.

    Cases included

    CaseTypeChecks
    refund-01typicalcontains "refund", max 120 words, judge: empathetic + next step
    track-01typicalregex "order number", judge: no invented tracking info
    edge-emptyedgejudge: asks a clarifying question
    adv-offtopicadversarialjudge: declines and redirects
    json-01edgevalid JSON with keys email, phone, chat

    CI gate

    python3 evals/run_evals.py --config evals/config.json --save-baseline
    python3 evals/run_evals.py --config evals/config.json --fail-under 0.8 --max-regression 0.05
    

    Next steps

    • Replace the starter cases with 20-50 real user inputs.
    • Run without --mock, then save the winner as your baseline.
    • Add ANTHROPIC_API_KEY as a repo secret to enable evals.yml.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Test prompts, skills and agents on the Claude API: compare versions and models in one report, and block pull requests when quality drops.

    Changing a prompt and hoping it got better is how regressions ship. This kit gives Claude a ready-made way to test prompts, skills and agents against real cases, compare versions and models side by side, and stop a pull request when quality drops.

    What you get

    FileWhat it does
    run_evals.pyRuns every case x prompt variant x model on the Claude API, async with retries; caches responses by hash so re-runs only pay for changes.
    Graderscontains, not_contains, regex, max_words, min_words, valid_json, json_keys, plus an LLM judge with a 1-5 rubric and pass threshold.
    ReportMarkdown + JSONL: pass rate, per-check pass rates, mean judge score, p50/p95 latency, tokens, cost (from prices you set), worst failures with the reason.
    Regression gate--save-baseline, --fail-under 0.85, --max-regression 0.05 exit non-zero for CI.
    --mockDry run with no API calls to check your setup.
    Starter filesExample cases (typical, edge, adversarial), two prompt variants, config, GitHub Actions workflow.

    Python 3.9+ and pip install anthropic.

    How it works

    1. Claude helps you define 3-6 checkable criteria and 20-50 cases from real inputs.
    2. List prompt variants and model IDs in config.json.
    3. python3 evals/run_evals.py --config evals/config.json
    4. Read the report, spot-check judge verdicts, fix the prompt, re-run.
    5. Save a baseline and add the workflow so every PR is checked.

    Limits

    LLM-judge scores are only as good as your rubric: spot-check them. Costs are estimates from the prices you enter. Uses the Claude API (not other providers) and needs your own API key.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Prompt & Skill Engineering.

    Requires Python 3.9+ and an Anthropic API key. Does not support multi-modal (image/video) evaluations. User must manually verify a subset of LLM judge verdicts for rubric accuracy.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 4 days ago

    Accounts
    Anthropic API key
    Software
    Python

    Permissions

    Terminal / Shell
    Read Files
    Write Files
    Network Access
    Listed4 days ago

    What's inside

    Frequently Asked Questions

    This skill provides a structured Python framework to replace 'vibe-based' testing with data-driven metrics. It prevents regressions by running multi-variant tests and enforcing quality gates before you deploy prompt changes to production.

    The harness is built for Python 3.9+ and specifically optimized for the Anthropic SDK. While it includes a GitHub Actions workflow for CI/CD, it can be integrated into any environment that supports Python execution and has access to an Anthropic API key.

    You receive the complete Python source code for the harness, a set of pre-configured grader templates (code-based and LLM-as-a-judge), a sample test case library, and a ready-to-use GitHub Actions workflow.

    Yes, the skill includes 'LLM-as-a-judge' grading where you can define qualitative rubrics. The harness uses a separate model call to score outputs based on your specific criteria, such as tone, helpfulness, or brand alignment.

    The harness features an intelligent caching system that hashes inputs and prompt variants. If you rerun a test without changing the prompt or input, it will pull from the cache instead of making a new API call, significantly reducing development costs.

    New in Prompt & Skill Engineering