More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Skill Eval: A/B Test Whether a Skill Helps

    1

    The public case behind this skill is one that scored 77% with it loaded and 97% without it (Nick Nisi, WorkOS).

    Free

    Security scanned
    Skill Eval: A/B Test Whether a Skill Helps

    Skill Eval: A/B Test Whether a Skill Helps

    Example session with this skill installed

    Is my release-notes skill worth keeping? Here are 3 tasks it should help with. Run the eval with 5 runs per arm.

    • Read your context and instructions
    • Compiled the eval

    Claude drafts a suite file with your 3 tasks and a hard-check rubric for each, avoiding prompts that name the skill's topic so the control arm does not trigger it. It runs the suite: 30 isolated task calls and 30 judge calls. The report shows each scenario's pass rate with and without the skill, for example 80% vs 40% on one task and 100% vs 100% on another, an overall line of 87% vs 67% (+20 points), the verdict KEEP, and a note that the judge is uncalibrated, so differences within 10 points may be judge error.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    About this skill

    A skill costs context on every call, and it can lower quality as easily as raise it. The public case behind this skill is one that scored 77% with it loaded and 97% without it (Nick Nisi, WorkOS). Without a measurement you cannot tell which kind you installed.

    This skill runs a suite of task prompts in two arms: once with the skill's SKILL.md injected as a system prompt and once without, several runs per arm. A judge model grades every output against the scenario's rubric, and the report shows the pass rate per scenario and overall with a verdict: KEEP at 10 or more points better, DELETE/REWRITE at 10 or more points worse, NEUTRAL in between. Each task call runs in an empty temp folder with no CLAUDE.md, MCP servers, tools, or hooks, so your own setup cannot leak the treatment into the control arm. Unparsable judge replies are excluded from both arms instead of counted as failures.

    The proof

    The repo README reports a run on Hardik Pandya's stop-slop writing skill: 3 scenarios, 5 runs per arm, Sonnet as task model and judge, rubrics fixed before the runs. With the skill: 100%. Without: 60%. Overall +40 points, verdict KEEP. The bundled qualitative-rigor example scored 100% in both arms on the same day, meaning current Sonnet already does what that skill asks, so it costs tokens for nothing. The README also notes that at 3 runs per arm one output flips a scenario by 33 points, which is why the skill tells you to treat per-scenario deltas under 5 runs as noise. A calibration script measures the judge's true-positive and true-negative rates against your hand labels and then corrects later reports for judge error.

    What you get

    SKILL.md with the rules for writing suites and reading verdicts; run-eval.sh, the runner; calibrate-judge.py, for judge calibration and corrected pass rates; two suites (qualitative-rigor.json, which runs against a bundled example skill, and stop-slop.json, which needs that skill installed separately); and the example skill itself.

    How to use

    1. Copy the skill folder to ~/.claude/skills/skill-eval/.
    2. Run the bundled example: ~/.claude/skills/skill-eval/run-eval.sh ~/.claude/skills/skill-eval/suites/qualitative-rigor.json --runs 3.
    3. Write a suite for your own skill with hard-check rubrics, run it, and read the overall line and verdict in the report path it prints.

    Source and license

    Free to use and MIT licensed. The full source, install steps, and issue tracker are on GitHub at https://github.com/robertmnyborg/skill-eval.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Requires the Claude Code CLI and real model calls for every run. Only the SKILL.md body is injected, not the files it links to; inline them into a test copy if they matter. The task model gets no tools, so for tool-heavy skills the rubric must grade the plan, not side effects. The stop-slop suite needs that skill installed at ~/.claude/skills/stop-slop. Until you calibrate the judge, a difference inside 10 points may be judge error.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free to use

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 8 days ago

    • Free with an Agensi account

    Software
    Claude CodeBashjqPython 3 (standard library only)

    Permissions

    File Scopes

    the
    skill
    folder's
    results/
    directory
    a
    temp
    sandbox
    per
    run
    SKILL.md
    under
    test
    (read
    only)

    Each run makes Claude Code CLI calls billed to your account. Task calls run with tools, MCP servers, and hooks disabled.

    Listed8 days ago

    What's inside

    Frequently Asked Questions

    No agent compares its own output with and without a skill on its own. This skill runs both arms in isolated sessions, has a separate judge grade them, and does the arithmetic.

    No. The runner calls the Claude Code CLI (`claude -p`) for every task and judge call, and installs as a Claude Code skill. It can evaluate any SKILL.md file, but it needs Claude Code to run.

    Scenarios times 2 arms times runs, plus the same number of judge calls. 3 scenarios at 3 runs per arm is 36 model calls. The SKILL.md suggests Sonnet for both task and judge.

    Until you calibrate it, you should not trust small differences, and every report says so. `calibrate-judge.py sample` draws 40 judged rows for you to label blind, and `score` reports the judge's true-positive and true-negative rates. After that, reports include corrected pass rates.

    On GitHub at https://github.com/robertmnyborg/skill-eval, under the MIT license. Every file in this package is there, plus the README and install steps.

    New in LLM Ops

    Free