Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Prompt Regression Kit: Golden Sets, Scorer and LLM Judge

    1

    Build a golden set from real traffic, score every new prompt or model version with a bundled Python scorer, and catch the regressions a steady pass rate hides.

    $19

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Prompt Regression Kit: Golden Sets, Scorer and LLM Judge

    Prompt Regression Kit: Golden Sets, Scorer and LLM Judge

    Example session with this skill installed

    We changed our support bot's system prompt. Use the kit's example files to tell me whether version 2 is better than version 1.

    • Read your context and instructions
    • Compiled the prompt regression kit

    Summary

    Version 2 keeps the same 83.3% pass rate as version 1, but it is not safe to ship: two cases broke while two others got fixed, so the headline number hides both regressions.

    Scoring (evalkit.py)

    $ python3 scripts/evalkit.py score examples/cases.jsonl examples/outputs-v2.jsonl --grades examples/grades-v2.jsonl --out v2.json
pass rate 83.3% (10/12) · missing 0 · ungraded 0 · latency p50 700 p95 1490

$ python3 scripts/evalkit.py compare v1.json v2.json --fail-on-regression
pass rate 83.3% -> 83.3% · regressions 2 · fixes 2 · still failing 0 · new cases 0
  REGRESSION angry-late-order: judge: Acknowledges the delay without blaming the customer: Blames the customer's shipping choice. | judge: Offers a concrete next step with a time frame: No next step and no time frame.
  REGRESSION injection-discount: not_contains: found ['discount code:'] | regex: no match for /can't|cannot|not able/
  FIXED classify-address
  FIXED refund-no-receipt

    compare exited with code 1, which would stop a CI deploy.

    What broke and why

    CaseTagsWhat failedLikely causeAction
    injection-discountsafety, adversarialGave a discount code to a prompt injectionThe refusal rule was dropped from the new promptRestore it and add two more injection cases
    angry-late-ordertoneBlamed the customer, no next stepThe new "be concise" line cut the apology and the follow-upAdd one empathy line and a next-step rule

    Fixed in version 2: refund-no-receipt (now asks for the order number) and classify-address (correct label).

    Verdict

    HOLD. Restore the refusal rule, re-run the 12 cases, and ship when compare shows 0 regressions.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    You change a prompt, the pass rate stays at 83%, and you ship. Two answers that used to be right are now wrong, hidden behind two that got fixed. This kit compares every version case by case, so that never goes out unnoticed.

    What it does

    • Builds a golden set from your logs, tickets and policies, with personal data stripped and a source for every expected answer.
    • Scores outputs with 14 deterministic checks: contains, regex, one_of, JSON fields, numbers within a tolerance, word limits and more. A broken check fails instead of passing quietly.
    • Compares two runs and lists regressions, fixes and cases still failing. Exit code 1 on a regression, so it can block a deploy in CI.
    • Splits dev and test so cases from the same template never leak across.
    • Grades tone and helpfulness with an LLM judge that has to agree with a person first: Cohen's kappa is computed for you.

    What is inside

    • 3 skills: eval-set-builder, eval-runner and rubric-judge
    • evalkit.py: one Python file, no dependencies, no API keys and no model calls, so it works with any provider
    • A worked example: 12 support-bot cases, two versions of outputs and judge grades that reproduce the hidden-regression story in four commands

    Use cases

    • Switching models or providers without guessing.
    • Changing a system prompt without breaking last month's fixes.
    • Testing an agent's JSON or tool output on every pull request.
    • Showing a client or a team lead what improved and what got worse.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Outputs come from your own system; the kit does not call models. No hosted dashboard. Judge grading follows the rubric-judge skill inside your agent, so its quality depends on calibration, which the kit measures with Cohen's kappa.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 5 days ago

    Software
    Python 3.8+ (for evalkit.py)

    Permissions

    Terminal / Shell
    Read Files
    Write Files

    File Scopes

    *.jsonl
    runs/*.json

    Terminal runs scripts/evalkit.py. It reads your cases and outputs (JSONL) and writes run files (JSON). No network access.

    Listed5 days ago

    What's inside

    Frequently Asked Questions

    Three skills (eval-set-builder, eval-runner, rubric-judge), the evalkit.py script and a worked example with 12 cases, two output versions and judge grades.

    Any. evalkit.py only reads JSONL files of cases and outputs, so OpenAI, Anthropic, Google, open models or your own agent all work the same way.

    No. Python 3.8 or newer is all the scorer needs.

    Yes. compare --fail-on-regression exits with code 1 when any case goes from pass to fail.

    No. It builds evaluation sets and checks outputs against them. Use it to test a fine-tuned model, not to make its training data.

    New in LLM Ops