Skill Bench: Catch Your Claude Skill's Bugs Before Buyers Do

A skill can pass one quick try and still fail on a repeat run, invent numbers, or make claims about itself that are not true. Skill Bench is a test method for catching that before your buyers do.

You point it at a skill. It helps you design a test set with real spread: different inputs, messy wording, traps where the skill's own rules must hold, and planted ground truth so answers are checked, not guessed. Nothing runs until you approve the set.

Every case runs with the skill and without it, several times. bench.py runs the code checks first (counts, lengths, numbers against a computed true value, forbidden terms, files produced). Then each answer goes to a grader in a shuffled, unlabelled packet, so the grader never knows which arm wrote it. Three grades: pass, pass with note, fail, each with a quoted reason.

Failures are classified as skill defect, test-set issue, harness artifact or grey area, and become a fix brief with the exact old and new wording, acceptance lines and a guard line for the fix's own risk. After you change the skill, the re-bench passes only if every acceptance line holds and nothing that passed before now fails.

It builds on Anthropic's skill-creator (its eval format, aggregator and review page) and adds the test design, blind grading, repeatability check and regression bar. It does not build skills or tune trigger descriptions; use skill-creator for that.

Compatibility

Best in Claude Code, or any Claude environment with sub-agents and Python 3.9+ (standard library only, no installs). Needs Anthropic's skill-creator skill for the benchmark summary and review page. In a plain claude.ai chat with no sub-agents it runs a labelled smoke test only, because one session cannot fairly write both arms. An optional API-batch route for large sets never spends without your go.

Permissions

• Read files: the skill you bench and your test set
• Write files: bench.py writes only to the bench workspace you point it at
• Delete files: only bench.py blind --force, which empties and rebuilds the packet folder you name
• Run Python: scripts/bench.py on your own files; a python check in your test set imports and runs a check module you write
• Sub-agents: one per run and one per grading packet
• Network: none, except the optional bench.py fresh --links, which checks that links in your own test set still load

Known limitations

• It is slower than skill-creator alone. On our own bench (v1.1, 6 synthetic cases) skill-creator took 37% less time and 7% fewer tokens per run.
• It does not build skills or tune trigger descriptions; it pairs with skill-creator.
• A bench only measures the cases you give it. In our own bench, two trap cases did not separate the arms: plain Claude passed them too.
• Blinding is partial: answers written with the skill can name bench.py, which hints at the arm.
• The grader is a model. Every grade quotes its evidence, and the method asks you to spot-check a few.
• Sub-agents can write outside their run folder (3 of 54 runs in our bench). Run it in a dedicated, empty folder.
• The number check reads a value written after its label in a sentence; values inside a table row go to the grader instead.
• Measured on one model family.

Claude is a trademark of Anthropic, PBC; this skill is not affiliated with or endorsed by Anthropic.