Mutation Score Gate

Your agent says all tests pass. This skill proves whether those tests would catch a real bug — by running real mutation testing tools, enforcing a kill-rate CI gate, and catching the test-tampering tricks agents use to fake green.

The Problem

Your AI coding agent just refactored the refund module. It reports all tests pass. You merge. Two days later a negative refund sails through to production — and the suite is still green. What happened: the agent wrote both the code AND the tests that check it, and somewhere along the way the boundary test got skipped, the assertion got loosened to toBeDefined, or a strict test was quietly swapped for a passing one. A controlled 2026 experiment found agent suites go full-green with vacuous assertions while bug-catching power sits near zero. A 591-session field study found 69 percent of agent done claims had no passing test run behind them. Coverage tools cannot see any of this. They record what ran, not what was verified.

What You Get

• Seven pre-claim substitution checks that run in seconds — skip scan, tautology scan, assertion-count diff, deleted-file scan, swallowed-error scan, vanished-test-name diff, empty-selection guard
• Real mutation runs per language — StrykerJS for JS and TS, mutmut and cosmic-ray for Python, pitest for Java, Stryker.NET for CSharp, cargo-mutants for Rust, advisory-plus-fuzz for Go
• A deterministic threshold gate script that parses the tool report artifact and fails the build below your kill-rate threshold, starting at 50 percent with a quarterly ramp plan
• Survivor triage that classifies every surviving mutant as real gap, equivalent mutant, no-coverage, or flaky — ranked by mutation type crossed with file criticality
• A smoke-gate procedure that proves the mutation tool observes mutations on one file before you burn hours on a full run
• A 3-run stability rule for agent-written suites, because a single green run swings benchmark scores several points even at temperature zero

Why This Beats Prompting It Yourself

Asking an agent is this test suite strong invites the same model that wrote the weak tests to grade its own homework — and no LLM judge configuration in published testing clears even 0.65 discrimination. This skill removes model judgment from the verdict entirely: a generated test either kills a mutant under the test runner or it does not. The pre-claim checks are shell and git commands, not opinions. The gate parses the tool JSON and XML report, not stdout. You get mechanical evidence — kill rate, named survivors, tamper flags — instead of a confident paragraph.

Example Session

You: My agent says all tests pass on the refund refactor. Verify before I merge.

Agent: Pre-claim checks against origin slash main: 2 FAILS. Check 1: describe dot skip added in refund dot test dot ts line 14. Check 6: vanished test name rejects negative amounts present in base, absent in head. The agent skipped the boundary test guarding negative refunds. Score: 2 of 7 checks failing. Stopping here — fix the tampering before trusting any green run. After you restore the test, Stryker scoped to src slash payments with threshold 50. Kill rate 62 percent: 34 killed, 21 survived, 3 no-coverage. Risk rank 1 of 21: conditional boundary amount less-than-or-equal zero versus amount less-than zero in validateRefundAmount line 96 — a real gap, killer test sketched asserting exact rejection of zero and negative amounts. Risk rank 2: arithmetic operator on fee rounding line 112 — real gap, second killer test sketched. Remaining 19: 2 equivalent with exclusion annotations, 17 low-risk string-literal noise deferred. Verdict: MERGEABLE once the 2 killer tests land. Totals: 1 substitution caught, 2 real bug-gaps closed, 0 false alarms.

You: Add a mutation gate to CI for the billing module.

Agent: Pytest plus mutmut lane detected. Smoke gate on billing slash pricing dot py: 4 killed, mechanism works. Scoped mutmut run on billing slash: kill rate 54 percent, threshold 50, gate PASSES. Wired check mutation gate dot py into the PR workflow scoped to changed files, documented the 50 percent start with quarterly 10 point ramp. Baseline recorded, 2 survivors queued as killer-test tasks for this sprint.

Use Cases

• Verifying an agent done claim before merging — run the 7 pre-claim checks plus a scoped mutation gate instead of trusting the summary
• Adding a mutation kill-rate gate to the PR pipeline on payments, auth, or data validation modules
• Triaging a demoralizing 30 percent score on a legacy module into a ranked fix list the team can work top-down
• Auditing a newly inherited suite that shows high coverage but ships frequent production bugs
• Gating a Go service honestly — advisory mutation signal plus real fuzz enforcement, with limits stated upfront

Known Limitations

Full mutation runs are slow on large codebases — scope to changed files per-PR and run full suites weekly. Equivalent mutants need human judgment to exclude; the skill classifies but you confirm. Go mutation tooling is alpha quality so Go mode is advisory-only by design. Requires a green baseline suite and a working toolchain per language before the gate means anything.

Tags: testing, mutation-testing, test-quality, ci-cd, code-quality, ai-agents Version: 1.0.0