Skill Evaluation Harness Builder

The problem

Skills are often trusted based on vibes rather than data. Without a formal regression suite, model updates or minor prompt edits can silently break core functionality without you noticing until a user complains.

What it does

• Analyzes a target SKILL.md to identify absolute promise-breaking behaviors.
• Generates five structured test cases including happy paths, defect detection, and boundary rule triggers.
• Constructs a "Probe" test case specifically designed to tempt the model into its known failure modes.
• Produces a standalone markdown harness with binary MECH (mechanical) and JUDGMENT grading criteria.
• Designs adversarial ground-truth sets for detection-type skills to measure false-positive rates.

Why this beats prompting it yourself

Most DIY evals suffer from "rubber-stamping" where the model passes everything that looks roughly correct. This skill enforces 0% partial credit and two-directional grading to catch both missed defects and fabricated errors that humans often overlook during manual testing.

Use cases

• Verify skill stability after swapping from GPT-4o to Claude 3.5 Sonnet.
• Run a regression suite after refactoring a skill's instruction set.
• Audit a skill's performance against "lazy" failure modes before marketplace listing.
• Debug why a skill is suddenly hallucinating or skipping safety guards.

Known limitations

Requires the user to provide or identify a reference document for any criteria involving subjective taste or brand tone.