Agent Skill Regression Tester Pro

Run a skill's regression suite in one command and catch a broken SKILL.md edit before it ships, with exit codes your CI already understands.

Free vs Pro

• Free: a methodology the agent follows by hand, ideal for a one-off pass on one skill. The free version has 16 installs as of 2026-10-11.
• Pro: the same method plus four executable scripts, with stable scenario files, a JSON report and exit codes for repeated runs, CI and multi-skill libraries.
• Everything in the free skill still applies; Pro adds tooling and removes nothing.

Upgrade Path

If you already use agent-skill-regression-tester, install Pro next to it and keep your existing scenarios. Run generate_harness.py on one skill, move your hand-written samples into sample_outputs/ named by scenario id, and run run_regression.py. Add it to CI once the exit codes are what you expect.

The Problem

SKILL.md files are prompts, so there is nothing to unit-test. The free agent-skill-regression-tester teaches an agent the method for fixing that, but the agent still hand-writes every scenario file, check script and report format each time, and rebuilds them slightly differently on the next skill. When a skill changes weekly, the harness gets skipped, and the first sign of a regression is a teammate saying the agent behaves differently.

What You Get

• contract_extractor.py reads a SKILL.md and writes contract.json: constraints, output contracts, triggers and decision rules, each with an id, a source line and extracted signals such as format strings, allowed values and length limits. It is regex plus a heading walk, so the same file gives the same contract every time.
• generate_harness.py turns that contract into a tests/ directory with scenario stubs, deterministic checks, a baseline folder per skill version and a shell wrapper. It writes scenarios.yaml when PyYAML is present and scenarios.json when it is not.
• run_regression.py runs the checks against your sample outputs, prints PASS, FAIL or WARN per scenario plus an OVERALL verdict, and writes test_report.json. Exit code 0 means no failures, 1 means a failure, 2 means the harness itself is broken.
• compare_versions.py diffs a baseline output folder against a candidate folder and writes diff_report.json with missing sections, changed counts and format breaks, flagging only the scenarios that need an LLM to judge meaning.
• Four pure-standard-library scripts: no pip installs, no network, no LLM calls, and every script has --help.
• Rules the extractor cannot check mechanically become manual checks that report WARN, so nothing is silently skipped.

Why This Beats Prompting It Yourself

Asking an agent to build a test harness gives you a different scenario format, check layout and report shape on every run, and nothing that behaves the same in CI. Pro fixes the format once: the report always has overall, total, passed, failed, warned and a per-scenario array, the exit codes are a contract, and contract extraction does not drift between runs. The agent spends its effort on the parts that need judgment, which are sharpening scenarios and reviewing the few semantic changes the scripts flag, instead of re-deriving plumbing.

Example Session

This is example output from the sample commit-message-generator skill that ships with Pro, not a real customer run.

• You ask: build a regression harness for commit-message-generator and tell me if my edit broke anything.
• The agent runs python3 scripts/contract_extractor.py examples/sample-target and gets 5 constraints, 4 output contracts, 3 triggers, 1 decision rules, including the format type(scope): description, six allowed types, a 10-72 char limit and a rule against Jira ticket ids in the title.
• The agent runs python3 scripts/generate_harness.py examples/sample-target --skill-name commit-message-generator and gets 10 scenario stubs, ids CS-001 to CS-010, with contract.json and scenarios.yaml written.
• After the skill edit, the agent produces sample outputs and runs python3 scripts/run_regression.py examples/sample-target.
• Result lines: CS-001 PASS, CS-002 PASS, CS-003 PASS, CS-004 PASS, CS-007 FAIL with not_matches_regex: pattern found in first_line, CS-008 FAIL with not_contains: 'misc' present, and four WARN scenarios marked manual.
• Verdict line: OVERALL: FAIL (total=10 passed=4 failed=2 warned=4), process exit code 1, so the edit is stopped before it ships.
• The agent fixes the instruction that let the ticket id through, regenerates the samples, and re-runs until the exit code is 0. It then reads the four WARN scenarios itself, because WARN means a human or LLM still has to look.

Use Cases

• Gate pull requests that touch a SKILL.md by running run_regression.py in CI and failing the job on exit code 1.
• Give every skill in a shared library the same scenario format and report shape with one command per skill.
• Check a skill update against its previous outputs with compare_versions.py and review only the scenarios it flags.
• Produce a test_report.json as pass/fail evidence before publishing a skill to a marketplace.
• Keep the baseline outputs for each skill version in snapshot_baselines/ so a regression can be traced to a version.

Known Limitations

The scripts check files, not live agents: sample outputs must be produced by your agent from the current SKILL.md, and stale samples prove nothing. Derived checks are stubs built from regexes, so a format string becomes a loose pattern and an allowed-values list becomes a "value appears" check that you should tighten. compare_versions.py is structural only and cannot say whether a changed answer is better or worse. Contract extraction recognises English constraint wording such as Never, Always and Do NOT use. A scenario file written as YAML cannot be run on a machine without PyYAML.