Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Llm Eval Mutation Harness

    1

    Prevent silent LLM prompt and RAG regressions using 30-case golden eval sets, mutation-checked deterministic graders, and token cost/latency release gates.

    $19

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    llm-eval-mutation-harness

    llm-eval-mutation-harness

    Example session with this skill installed

    We want to rewrite our extraction system prompt in src/prompts/extractor.ts and switch from temperature 0.2 to 0.0. Build a golden eval harness and verify there are no regressions.

    • Read your context and instructions
    • Compiled the llm-eval-mutation-harness

    LLM Pipeline Evaluation & Mutation Receipt

    MetricBaseline (v1.2, temp=0.2)Candidate (v1.3, temp=0.0)DeltaGate Verdict
    Golden Pass Rate (32 cases)29/32 (90.6%)32/32 (100.0%)+9.4%PASS
    Critical Schema Regressions000PASS
    Avg Prompt + Completion Tokens1,8401,390-24.5%PASS
    p95 Latency1,420 ms1,080 ms-340 msPASS
    Mutation Check (Guard Removed)n/aFailed 6/32 (case-04..09)Grader VerifiedPASS

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Changing a system prompt, switching models, tweaking temperature, or refactoring a tool schema without running a quantitative evaluation harness is the #1 cause of silent production regressions in AI applications.

    llm-eval-mutation-harness equips Claude Code, Cursor, Windsurf, and Copilot with a strict, reproducible evaluation engineering workflow:

    • 30-Case Golden Eval Dataset Standard: Automatically scaffolds stratified test suites covering happy paths, adversarial prompt injections, malformed JSON boundaries, Unicode/empty inputs, and multi-step tool calls.
    • Mutation-Checked Graders: Proves your evaluation assertions actually catch regressions by mutating the prompt guard or schema validator and verifying the eval suite fails before trusting a green pass rate.
    • Single-Variable A/B Delta Reporting: Enforces changing one variable at a time (prompt, model, temperature, or parser) and generates a per-case regression matrix alongside p50/p95 latency and token-cost deltas.
    • Zero-Regression Ship Gate: Blocks deployment if any critical safety/schema case regresses or if token cost per task increases by >15% without a matching accuracy gain.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Prompt & Skill Engineering.

    Requires initial manual baseline recording. Performance gates for cost and latency require a minimum of 30 cases for statistical significance.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 10 days ago

    Permissions

    No special permissions declared or detected

    Listed10 days ago

    What's inside

    Frequently Asked Questions

    This skill automates the creation of a 30-case 'golden set' and uses mutation testing—deliberately injecting errors—to verify that your automated grader is actually capable of catching regressions before they hit production.

    The harness is framework-agnostic and works with any LLM provider or RAG pipeline that can output to .jsonl, making it compatible with OpenAI, Anthropic, local models, and custom orchestration layers.

    You receive the full evaluation harness logic, templates for stratified dataset generation, the mutation testing suite, and a workflow guide for integrating these checks into your CI/CD pipeline.

    The setup involves a five-step initialization process starting with the generation of a `golden.jsonl` file and capturing a performance baseline; it is designed to be integrated into your existing development environment in under 15 minutes.

    While it works with smaller sets, a minimum of 30 stratified cases is required to ensure statistical significance for the cost and latency gating features, preventing outliers from triggering false alarms.

    Updates are delivered through the marketplace and typically include new adversarial edge-case templates and optimizations for the mutation testing logic to keep up with evolving LLM behavior.

    New in Prompt & Skill Engineering