Fix Failing Tests Without Skipping or Weakening Them

The problem

Ask an AI agent to "make the tests pass" and it often will, by editing the expected value, wrapping a flaky test in a retry, or adding .skip. The suite turns green and the bug ships. Tests are your specification; silencing them isn't fixing them.

What this skill does

It treats every failing test as evidence, and works out why it fails before changing anything. Each failure gets exactly one category:

When the evidence is unclear, it assumes the test is right.

When a test looks outdated, you choose:

• yes: it updates the test's expected value as shown.
• no: the test is right, so the source should change instead. It doesn't edit anything yet. It shows the evidence that the source change looked deliberate, the exact source diff, a warning that this reverses an intentional-looking change, and what else depends on the value. Then it asks again: yes to apply that source change, or leave it.
• leave it: nothing changes; the test is reported as unresolved and the suite stays failing.

Each edit needs its own "yes" to its own question, and the report says which approval covered which change.

Hard rules. It never deletes or skips a test, loosens an assertion, changes an expected value, updates snapshots, or adds retries or sleeps, unless you explicitly approve after it explains why the test itself is wrong. It never installs packages without asking, and never opens .env or other secrets.

Who it's for

Anyone who's had a suite break after a refactor, merge or dependency bump, and wants it fixed properly. Also teams who want an agent they can point at CI failures without worrying it will quietly hollow out their tests.

What the agent does, step by step

1. Runs your project's own test command and groups failures by shared root cause.
2. Collects evidence for each one: the failure, the code at the trace, recent git history of the test and the source, docs and changelog, isolated and repeated runs.
3. Assigns a category and fixes environment issues first, then code bugs, then flaky tests. For outdated tests, it asks (yes / no / leave it) and edits nothing for that test until you answer.
4. Re-runs the full suite (at most three passes through steps 2–4, so it won't thrash).
5. Reports what was fixed and how, what needs your decision, what's unresolved, and a test integrity line confirming nothing was weakened.

Example output

From a real run on a broken TypeScript + Vitest project (the first report, before any answer; verbatim, shortened after the question). The full run, including the report after "approved", is in the example file.

## Test suite: still failing, needs your decision

**Before:** the test file couldn't load (0 of 9 tests ran) · **After:** 1 failed / 9 total
**Command:** `npm test` (`vitest run`)

### Fixed
| Test(s) | Category | Root cause | Fix |
|---|---|---|---|
| Whole file (`Cannot find package '@/shipping'`) | Environment/setup | Commit `e649fbb` ("Tidy vitest config") removed the `@` alias from the vitest config. `tsconfig.json` still maps `@/*` to `src/*`. | `vitest.config.ts:5`: put the `resolve.alias` back and kept the new `test.include` |
| `freeShippingEligible › members qualify from exactly $25` | Code bug | The same commit changed `orderTotal >= 25` to `> 25`. That contradicts the docstring, which says members get free shipping "from $25". | `src/shipping.ts:38`: back to `>= 25` |
| `shippingCost › prices any allowed weight` | Flaky | The test picked a random weight from 0.1 to 50.1 kg, so about 40% of runs used a weight over 30 kg, which the code rejects. | `test/shipping.test.ts:21`: the random weight is replaced with fixed weights 0.1, 1, 2, 2.1, 15, 29.9 and 30 kg, which cover the whole allowed range and its edges. The check (`toBeGreaterThan(0)`) is unchanged. It passed 5 out of 5 runs on its own. |

### Needs your decision
| Test | Why I think the test is wrong | Proposed change |
|---|---|---|
| `shippingCost › charges the international base rate` | The test expects 15 and the code returns 18. Commit `d8f0192` ("Raise international base rate to $18 (carrier contract renewal)") and the `CHANGELOG.md` 1.1.0 entry both show the $18 rate was deliberate. | `test/shipping.test.ts:9`: `.toBe(15)` → `.toBe(18)` |

Reply **yes** to update the test as shown, **no** if the test is right and the source should change instead (I'll propose that change and ask again before editing anything), or **leave it** to change nothing.

Supported

Limitations (honest)

• It fixes failing tests; it doesn't write new tests or improve coverage.
• It won't change an expected value on its own, so a suite with outdated tests ends "still failing, needs your decision" until you answer. That's deliberate.
• Answering no doesn't revert your source in one step. It proposes the source change and asks again, so fixing a test the "no" way takes two answers.
• Fixes outside the repo (installing packages, runtime versions, services, CI secrets) are handed to you as exact commands rather than done automatically.
• If it can't run your test command at all, it says so and reports what it could determine, rather than guessing results.

Tags

failing-tests, debugging, flaky-tests, ci, jest, vitest, pytest, test-maintenance