More screenshots
Works with the AI Agents you already use
Skill Eval: A/B Test Whether a Skill Helps
The public case behind this skill is one that scored 77% with it loaded and 97% without it (Nick Nisi, WorkOS).
Free
Skill Eval: A/B Test Whether a Skill Helps
Example session with this skill installed
Is my release-notes skill worth keeping? Here are 3 tasks it should help with. Run the eval with 5 runs per arm.
- Read your context and instructions
- Compiled the eval
Claude drafts a suite file with your 3 tasks and a hard-check rubric for each, avoiding prompts that name the skill's topic so the control arm does not trigger it. It runs the suite: 30 isolated task calls and 30 judge calls. The report shows each scenario's pass rate with and without the skill, for example 80% vs 40% on one task and 100% vs 100% on another, an overall line of 87% vs 67% (+20 points), the verdict KEEP, and a note that the judge is uncalibrated, so differences within 10 points may be judge error.
Connects securely to your tools. The creator never sees your data.
About this skill
About this skill
A skill costs context on every call, and it can lower quality as easily as raise it. The public case behind this skill is one that scored 77% with it loaded and 97% without it (Nick Nisi, WorkOS). Without a measurement you cannot tell which kind you installed.
This skill runs a suite of task prompts in two arms: once with the skill's SKILL.md injected as a system prompt and once without, several runs per arm. A judge model grades every output against the scenario's rubric, and the report shows the pass rate per scenario and overall with a verdict: KEEP at 10 or more points better, DELETE/REWRITE at 10 or more points worse, NEUTRAL in between. Each task call runs in an empty temp folder with no CLAUDE.md, MCP servers, tools, or hooks, so your own setup cannot leak the treatment into the control arm. Unparsable judge replies are excluded from both arms instead of counted as failures.
The proof
The repo README reports a run on Hardik Pandya's stop-slop writing skill: 3 scenarios, 5 runs per arm, Sonnet as task model and judge, rubrics fixed before the runs. With the skill: 100%. Without: 60%. Overall +40 points, verdict KEEP. The bundled qualitative-rigor example scored 100% in both arms on the same day, meaning current Sonnet already does what that skill asks, so it costs tokens for nothing. The README also notes that at 3 runs per arm one output flips a scenario by 33 points, which is why the skill tells you to treat per-scenario deltas under 5 runs as noise. A calibration script measures the judge's true-positive and true-negative rates against your hand labels and then corrects later reports for judge error.
What you get
SKILL.md with the rules for writing suites and reading verdicts; run-eval.sh, the runner; calibrate-judge.py, for judge calibration and corrected pass rates; two suites (qualitative-rigor.json, which runs against a bundled example skill, and stop-slop.json, which needs that skill installed separately); and the example skill itself.
How to use
- Copy the skill folder to
~/.claude/skills/skill-eval/. - Run the bundled example:
~/.claude/skills/skill-eval/run-eval.sh ~/.claude/skills/skill-eval/suites/qualitative-rigor.json --runs 3. - Write a suite for your own skill with hard-check rubrics, run it, and read the overall line and verdict in the report path it prints.
Source and license
Free to use and MIT licensed. The full source, install steps, and issue tracker are on GitHub at https://github.com/robertmnyborg/skill-eval.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
Requires the Claude Code CLI and real model calls for every run. Only the SKILL.md body is injected, not the files it links to; inline them into a test copy if they matter. The task model gets no tools, so for tool-heavy skills the rubric must grade the plan, not side effects. The stop-slop suite needs that skill installed at ~/.claude/skills/stop-slop. Until you calibrate the judge, a difference inside 10 points may be judge error.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free to use
No account required to browse
Trust & safety
Security scan
Security scan passed 8 days ago
- Free with an Agensi account
- Software
- Claude CodeBashjq
Python 3 (standard library only)
Permissions
File Scopes
Each run makes Claude Code CLI calls billed to your account. Task calls run with tools, MCP servers, and hooks disabled.