Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Flaky Test Hunter: reproduce, classify, fix

    1

    The usual responses make things worse: hit "re-run", add a `sleep`, turn on suite-wide retries, or quietly delete the test.

    $9

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Flaky Test Hunter: reproduce, classify, fix

    Flaky Test Hunter: reproduce, classify, fix

    Example session with this skill installed

    CI fails on tests/test_invoice.py::test_due_date maybe once a week and passes on re-run. Find out why and fix it.

    • Read your context and instructions
    • Compiled the flaky test hunter

    Example session (illustrative).

    1. Reproduce in isolation.
    pytest tests/test_invoice.py::test_due_date --count=300 -x -p no:randomly gave 0/300 failures. The test does not fail on its own.

    2. Reproduce in shuffled suite order.
    pytest tests/ with pytest-randomly installed, run 20 times, failed twice: seeds 48213 and 9917. pytest tests/ --randomly-seed=48213 fails every time, so this is order dependence.

    3. Find the polluter.
    detect-test-pollution --failing-test tests/test_invoice.py::test_due_date --tests tests/ narrowed it to tests/test_settings.py::test_override_timezone.

    4. Root cause.
    test_override_timezone sets os.environ["TZ"] = "Asia/Tokyo" and calls time.tzset(), and never restores either. test_due_date computes "end of today" with naive datetime.now(), so after the polluter runs the date shifts by a day.

    5. Fix. Two changes:

    • test_override_timezone now uses monkeypatch.setenv("TZ", ...) plus a fixture that calls time.tzset() again on teardown.
    • test_due_date freezes time with @freeze_time("2025-01-31 23:30:00") and asserts the month-end rollover explicitly.

    Side finding: invoice.due_date() uses naive local time in production code. Flagged as a real bug, since invoices issued near midnight UTC get the wrong due date.

    6. Verify.
    pytest tests/ --randomly-seed=48213 passes. 30 shuffled suite runs gave 0/30 failures. Isolated: 0/300.

    Test:      tests/test_invoice.py::test_due_date
    Status:    FIXED
    Repro:     pytest tests/ --randomly-seed=48213
    Before:    2/20 shuffled suites failed   After: 0/30
    Class:     order dependence + time/timezone
    Cause:     test_override_timezone leaked TZ; test relied on wall clock
    Fix:       monkeypatch + tzset teardown; freeze_time in test_due_date
    Follow-up: production bug, due_date() uses naive local time (issue drafted)
    

    Connects securely to your tools. The creator never sees your data.

    About this skill

    The problem. Flaky tests erode trust in CI. The usual responses make things worse: hit "re-run", add a sleep, turn on suite-wide retries, or quietly delete the test. Coding agents are especially prone to this. They see one green run and declare victory.

    What the agent does with this skill.

    • Reproduces with real numbers. Uses the rule of three (about 3/p clean runs for 95% confidence) and the exact repeat commands for each stack: pytest-repeat --count, serial Jest loops, go test -count=N -race -cpu 1,2,8, and cargo test loops.
    • Diagnoses from the failure pattern. Checks whether the test fails alone, only in suite order, only in parallel, or only in CI. Each pattern points to a different cause class.
    • Runs in random order and keeps the seed. Uses pytest-randomly, jest --randomize --showSeed, vitest --sequence.shuffle, go test -shuffle=on, and nightly libtest --shuffle, then replays the failing seed. For order-dependent Python tests it finds the polluting test with detect-test-pollution.
    • Applies the fix for the cause class. For example: freeze or inject the clock, seed RNGs and log the seed, wait on events instead of sleeping, use per-test temp dirs, bind port 0, restore env vars, reset state in teardown, and catch leaks with --detectOpenHandles or goleak.
    • Quarantines only with a paper trail. The test is skipped with a reason and an issue link, it stays visible, and its owner is named. It never deletes tests or retries until green.
    • Writes a short report per test. Each report has the repro command, the before and after failure counts, the cause class, the fix, and any follow-up.

    Who it's for. Teams whose CI is "red, re-run, green", maintainers triaging intermittent failures, and anyone letting coding agents touch test suites.

    Requirements. Any agent that supports SKILL.md (Claude Code, Cursor, Codex CLI, and others) with shell access to run the project's tests. Optional helpers the agent may suggest installing: pytest-repeat, pytest-randomly, detect-test-pollution, goleak, serial_test. Rust order shuffling needs a nightly toolchain. It works on its own and pairs well with AgentHarness's no-network sandbox.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Testing & Debugging.

    Supports JS/TS, Python, Go, and Rust only. Extremely rare flakes may require execution times beyond standard agent timeouts to hit statistical significance.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 17 days ago

    Software
    Project test runnerPythonjestNode.jsvitest
    Optional
    pytest-repeatpytest-randomlydetect-test-pollutiongoleak

    Works with pytest, Jest, Vitest, go test or cargo test.

    Permissions

    Terminal / Shell
    Read Files
    Environment Variables
    Listed17 days ago

    What's inside

    Frequently Asked Questions

    No. The skill explicitly forbids retries, `sleep` calls and loosened timeouts as fixes. Suite-wide reruns (pytest `--reruns`, `jest.retryTimes`, nextest `--retries`) are treated only as a way to measure flakiness.

    Python (pytest), JavaScript/TypeScript (Jest and Vitest), Go (`go test`) and Rust (`cargo test`). The classification and quarantine policy apply to any stack.

    No, but some cause classes go faster with helpers such as pytest-repeat, pytest-randomly, detect-test-pollution, goleak or serial_test. The agent asks before adding dependencies. Shuffled order in Rust needs a nightly toolchain, because `--shuffle` is an unstable libtest flag.

    It is quarantined. The test is skipped in code with a reason and a tracking-issue link (`pytest.mark.skip(reason=...)`, `t.Skip`, `#[ignore = "..."]`, `test.skip`), and it is listed in the report. It is never deleted or silently commented out.

    No. The skill works on its own in any repo. If you use AgentHarness, its no-network sandbox makes network-caused flakes show up every time instead of now and then, and this skill tells the agent how to fix them properly.

    New in Testing & Debugging