Evidence-First Debugging
Evidence-first debugging: reproduce, measure flakiness, find the root cause, fix it with a regression test.
Free
Works with the AI Agents you already use
Claude Code
Cursor
Codex CLI
Muse
Dots+22 moreClaude CodeCursorCodex CLIMuseDots
OpenClaw
Manus
Grok
Grok Bot+18 agentsEvidence-First Debugging
Example session with this skill installed
test_get_user_returns_record fails now and then in CI with a TypeError. Find out why and fix it properly.
- Read your context and instructions
- Compiled the evidence-first debugging
Real session on a small Python repo. The script output is pasted as-is.
Summary
The test is flaky (6 of 20 runs fail). The cause is randomness in get_user, added in the most recent commit to the file. It's fixed at the cause, with a regression test, and now 0 of 20 runs fail.
1. Reproduce and measure
$ python3 scripts/repro.py -n 20 -- python3 -m unittest tests.test_cache
python3 -m unittest tests.test_cache: 6/20 runs failed (FLAKY)
×6 FAILED (errors=1)
hint: look for shared state, ordering, time, randomness or leftover files between runs
2. Read the evidence
$ python3 scripts/test_summary.py -- python3 -m unittest tests.test_cache
FAILED (exit 1) FAILED (errors=1)
ERROR tests.test_cache.T.test_get_user_returns_record
File "tests/test_cache.py", line 7, in test_get_user_returns_record
TypeError: 'NoneType' object is not subscriptable
get_user sometimes returns None.
3. Check recent history
$ python3 scripts/recent_changes.py app/cache.py
recent commits touching app/cache.py:
f8174b0 2026-10-18 cache: skip refetch for speed
a8e0514 2026-10-18 add user cache
4. Root cause
| Hypothesis | Evidence | Verdict |
|---|---|---|
| Shared state between tests | Only one test in the module, and it still fails 30% of runs | Ruled out |
| Randomness in the cache | app/cache.py calls random.random() to decide whether to return the entry or None | Confirmed |
5. Fix at the cause, plus a regression test
_cache = {}
def get_user(uid):
if uid not in _cache:
_cache[uid] = {"id": uid, "name": f"user{uid}"}
return _cache[uid]
def test_cached_user_is_always_returned(self):
for _ in range(200):
self.assertIsNotNone(get_user(2))
6. Verify
$ python3 scripts/repro.py -n 20 -- python3 -m unittest tests.test_cache
python3 -m unittest tests.test_cache: 0/20 runs failed (never fails)
Next steps: run the full suite in CI, and check other "speed" commits for similar shortcuts.
Connects securely to your tools. The creator never sees your data.
About this skill
Debugging by guessing leads to regression loops and masked symptoms. This skill enforces an evidence-first methodology to resolve crashes, hangs, and flaky tests by isolating the root cause before writing a single line of production code. It provides a structured workflow to reproduce failures, analyze error signatures, and verify fixes with statistical rigor.
What it does
- Flakiness analysis uses repetition scripts to determine if a failure is deterministic or intermittent.
- Error signatures groups the distinct failures seen across repeated runs, so you can tell one bug from several.
- Change tracking lists recent local commits that touched a file or a piece of text, to check whether a recent edit caused the failure.
- Hypothesis logging maintains a record of attempted experiments and their results to prevent circular debugging.
- Regression testing ensures every fix includes a test case that failed prior to the code change.
How it works
- Reproduce the failure using
repro.pyto establish a baseline failure rate and capture the exact stack trace. - Isolate the problem by bisecting input data or using
recent_changes.pyto identify relevant recent edits. - Hypothesize and test specific causes using prints or assertions, logging each result to confirm or rule out theories.
- Fix and Verify by applying the root cause solution and running
test_summary.pyor high-iteration reproduction scripts.
Frameworks & tools
The skill utilizes Python 3 standard library scripts for reproduction and change analysis. It integrates with standard test runners like pytest and unittest.
Why this beats prompting it yourself
Standard prompts often jump to code fixes based on symptoms, which masks bugs rather than solving them. This skill enforces a strict sequence of reproduction and evidence gathering, ensuring you don't waste time on "lucky" fixes that break later.
Use cases
- Debugging intermittent CI failures that only happen 5% of the time.
- Identifying the root cause of a silent hang in a data pipeline.
- Checking whether a recent commit to a file introduced a regression.
- Cleaning up complex error logs to find the first meaningful failure point.
Known limitations
Scripts require a Python 3 environment. It is designed for logic failures and crashes rather than visual UI inconsistencies or CSS debugging.
Free and open source (Apache-2.0), by the author of rosetta-js, a terminal coding agent for open-weight models.
How to use
Skills on Agensi run inside your AI agent. Copy the prompt, paste it into your agent, and it connects to Agensi and installs this skill for you. If a payment is needed, your agent shows it to you first.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill passes our trust checks below.
Security scanned
Passed our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free forever
No account required to browse
Trust & safety
Security scanned
Verified clean 2 days ago
- Free to download with an account
Requires
git