Test Tamper Detector: Catch Agents Gaming Your Tests

    1

    **The pain.** You ask a coding agent to make the failing tests pass.

    $12

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Test Tamper Detector: Catch Agents Gaming Your Tests

    Example session with this skill installed

    My coding agent says it fixed the pricing bugs and "all tests pass". Here is its diff. Did it actually fix the code, or did it change the tests?

    • Read your context and instructions
    • Compiled the test tamper detector

    Test Tamper Report

    Input: samples/agent-change.diff.txt (DEMO)
    Files: 7 (test 4, source 1, config/CI 2, snapshot 0)
    Findings: 11 high, 5 medium, 0 low, 0 accepted by a human

    Verdict: TAMPER_LIKELY - at least one change can make failing tests pass without fixing the code. Review before merging.

    #SeverityRuleLocationEvidence
    1HIGHCI_FAILURE_MASKED.github/workflows/ci.yml:10continue-on-error: true
    2HIGHTESTS_EXCLUDED_IN_CONFIGpyproject.toml:2addopts = "-q --deselect tests/test_pricing.py::test_bulk_price_threshold"
    3HIGHCOVERAGE_THRESHOLD_LOWEREDpyproject.toml:5fail_under = 70 <br>fail_under: 90 -> 70
    4HIGHSPECIAL_CASE_MATCHES_TESTsrc/pricing.py:8if total == 19.99 and percent == 15: <br>input 19.99 and output 16.99 appear together in an existing test
    5HIGHSKIP_ADDEDtests/test_pricing.py:9@pytest.mark.skip(reason="flaky on CI")
    6HIGHTEST_REMOVEDtests/test_pricing.py:14 (old)def test_discount_full_price(): <br>test 'test_discount_full_price' removed
    7HIGHASSERTION_WEAKENEDtests/test_pricing.py:22assert result <br>was: assert result == 334.80
    8HIGHTEST_FILE_DELETEDtests/test_tax.py:1 (old)deleted file tests/test_tax.py <br>2 test(s) removed with the file: test_tax_after_discount, test_tax_zero_rate
    9HIGHFOCUS_ADDEDweb/cart.test.ts:5it.only("sums line items", () => {
    10HIGHASSERTION_WEAKENEDweb/cart.test.ts:11expect(cartTotal([{ price: 10, qty: 1 }], "SAVE10")).toBeTruthy(); <br>was: expect(cartTotal([{ price: 10, qty: 1 }], "SAVE10")).toEqual(9);
    11HIGHEXCEPTION_SWALLOWED_IN_TESTweb/cart.test.ts:12} catch (e) {}
    12MEDIUMHARDCODED_RETURN_MATCHES_TESTsrc/pricing.py:9return 16.99 <br>tests use: 16.99
    13MEDIUMCOVERAGE_EXCLUSION_ADDEDsrc/pricing.py:20def _debug_dump(rows): # pragma: no cover (tamper-ok: local debugging helper, never shipped) <br>author note (not counted as acceptance): local debugging helper, never shipped
    14MEDIUMASSERTIONS_DROPPEDtests/test_pricing.py:15 (old)assert apply_discount(149.99, 0) == 149.99 <br>6 assertion line(s) removed, 4 added across test files (net -2)
    15MEDIUMTOLERANCE_LOOSENEDtests/test_pricing.py:16assert bulk_price(2.50, 50) == pytest.approx(112.50, rel=0.05) <br>explicit tolerance added to approx()
    16MEDIUMEXPECTED_VALUE_CHANGEDtests/test_pricing.py:17assert bulk_price(2.50, 49) == 122.25 <br>was: assert bulk_price(2.50, 49) == 122.50

    Paste this to the agent

    Before this change can merge, restore each item below.
    Do not edit, skip or weaken tests to make them pass, and do not add notes that mark items as approved.
    If you believe an item is a real requirement change, reply with the ticket or spec that says so; a human decides.
    1. .github/workflows/ci.yml:10 [CI_FAILURE_MASKED] Remove the mask so a failing step fails the build.
    2. pyproject.toml:2 [TESTS_EXCLUDED_IN_CONFIG] Which tests stop running, and why?
    3. pyproject.toml:5 [COVERAGE_THRESHOLD_LOWERED] Restore the threshold and add tests instead.
    4. src/pricing.py:8 [SPECIAL_CASE_MATCHES_TEST] Is this a general rule, or a hard-coded answer for the test input?
    5. tests/test_pricing.py:9 [SKIP_ADDED] Remove the skip and fix the failure, or link the issue that tracks it.
    ...
    

    Connects securely to your tools. The creator never sees your data.

    About this skill

    The pain. You ask a coding agent to make the failing tests pass. It comes back with "all tests pass". Sometimes it fixed the code. Sometimes it added .skip, turned toEqual(9) into toBeTruthy(), deleted the test file, set continue-on-error: true, dropped the coverage gate from 90 to 70, or wrote if total == 19.99: return 16.99 so one test goes green. Both versions look the same in the agent's summary, and the second kind is easy to miss in a long diff.

    The outcome. A verdict on the diff (TAMPER_LIKELY, REVIEW or CLEAN), every suspicious change with file:line, what the line was before, why it matters, and a numbered "paste this to the agent" block that tells it exactly what to restore. Exit code 1 on TAMPER_LIKELY, so it also works as a CI or pre-commit gate.

    The mechanism. A Python script (standard library only, no network, never edits your files) parses the unified diff and runs 37 rules across pytest/unittest, Jest/Vitest/Mocha/Playwright, Go/testify, JUnit/AssertJ, xUnit/NUnit/MSTest and RSpec, plus runner, coverage, build and CI configs and .gitattributes. It pairs each removed assertion with its replacement and ranks their strength (exact value > shape > truthiness). With --git it also searches the test files at the base revision, so a hard-coded answer in source is caught even when no test file changed. Notes written inside the diff (tamper-ok: ...) are shown but never accepted; only a human accepts a finding, on the command line. The agent then judges each finding as confirmed, legitimate or unclear.

    The limits. It reads the diff, not your intent: a real requirement change triggers the same patterns, which is why findings are questions and acceptance is explicit. It does not run tests, prove they pass, find bugs, or judge whether the original tests were good. Small integers (if x == 7) are not matched, to keep noise down. Works better with Git (reads the diff and base tests itself) and the GitHub CLI (gh pr diff); pairs with a run receipt such as Proof-of-Done Receipts, which proves the tests ran.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Testing & Debugging.

    Reads the diff, not intent: a legitimate requirement change triggers the same patterns, so each finding needs a human or agent judgment and acceptance is explicit (--accept / --allow). Does not run tests or check that they pass, does not find bugs, and does not judge whether the original tests were good. Assertion pairing is line-based; a large test-file rewrite shows up as dropped or renamed tests and needs a human read. Hard-coded-answer checks ignore small integers (if x == 7) and word-only values give at most MEDIUM. Skips applied by command-line flags or plugins outside the repo are not seen. Dependency manifests and lockfiles are skipped.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 5 days ago

    Software
    Python 3.8+Terminal access
    Optional
    GitGitHub CLI

    Git lets the script read the diff and the base test files itself (--git). Without Git, pass a saved diff with --diff. On Windows use python or py if python3 is missing.

    Permissions

    File Scopes

    Reads
    the
    git
    diff
    or
    a
    saved
    file
    With
    --git
    --tests-from
    reads
    existing
    test
    files
    Writes
    report
    only
    when
    --out
    is
    given

    No network calls. Never edits, reverts or fixes your code or tests.

    Listed5 days ago

    What's inside

    Frequently Asked Questions

    No. It reads the diff only (plus existing test files with --git or --tests-from). It tells you whether the tests were changed to pass, not whether they pass. Pair it with a run receipt for that.

    It will still be flagged, because it looks the same in a diff. Label it legitimate with the ticket that says so, and pass --accept RULE:PATH:LINE (or --allow RULE:GLOB) on the next run. Notes written inside the diff, such as 'tamper-ok:', never accept anything, because the change under review could have written them.

    Python (pytest, unittest, mock), JavaScript/TypeScript (Jest, Vitest, Mocha/Chai, Playwright), Go (testing, testify), Java/Kotlin (JUnit 5, AssertJ), C# (xUnit, NUnit, MSTest) and Ruby (RSpec), plus pyproject, setup.cfg, package.json, runner and coverage configs, codecov, Maven, Gradle, GitHub Actions, GitLab CI, Makefiles and .gitattributes.

    Yes. It exits 1 on TAMPER_LIKELY by default (--fail-on medium to also fail on REVIEW). SKILL.md has a GitHub Actions step; it needs fetch-depth: 0 so git can find the merge base.

    No. The script uses the Python standard library only, makes no network calls, needs no keys, and never edits your files. It writes a report file only when you pass --out.

    New in Testing & Debugging