- Home
- Skills
- Testing & Debugging
- Agent Trace to Regression Tests: Reproduce & Verify
Agent Trace to Regression Tests: Reproduce & Verify
Turn a failed Agent trace into executable pytest checks. Reproduce tool-call errors, test a revised agent through your local adapter, and keep a standalone test plus HTML and JUnit reports.
$1
Works with the AI Agents you already use
Agent Trace to Regression Tests: Reproduce & Verify
Example session with this skill installed
Use the regression runner with a real Codex Agent and a local stock lookup tool. Reproduce an obsolete storefront SKU mapping, fix only the mapping, then rerun the same request and assertions. Replay the recorded failure too. Keep all model events and tool execution receipts, and report the limits of this controlled test.
- Read your context and instructions
- Compiled the agent trace to
- Generated the document
Real Agent run: failure reproduced, fix verified
Executed on 2026-11-12 using Codex CLI 0.166.0 with existing login, Python 3.11.16 and pytest 8.3.3. The existing runner's command mode started two fresh model-backed Agent turns in read-only sandboxes. The inventory tool and stock values were controlled local fixtures.
| Run | Actual tool call | Actual Agent answer | Check result | pytest exit |
|---|---|---|---|---|
| Real Agent with obsolete route | lookup_stock(sku=SKU-24) | SKU-24 available=0 | 2/4 passed; case FAILED | 1 |
| Replay of that recorded failure | lookup_stock(sku=SKU-24) | SKU-24 available=0 | 2/4 passed; case FAILED | 1 |
| Real Agent after route correction | lookup_stock(sku=SKU-42) | SKU-42 available=17 | 4/4 passed; case PASSED | 0 |
Defect and repair
The storefront contract defined summer-chair as SKU-42 with 17 available units. The obsolete application route mapped it to SKU-24. The Agent read that route, executed the stock tool, and returned the wrong product's stock. The tests caught both the wrong SKU argument and wrong exact output. The repair changed only the route value from SKU-24 to SKU-42. The fixed Agent read the corrected route, executed the same tool and passed all four assertions.
The four assertions required lookup_stock, limited it to one call, checked every call's sku argument, and checked the exact final response. The prompt, request, tool code, stock data, CLI options and suite expectations were unchanged. Two distinct completed CLI turns and token usage records were retained. Tool calls were normalized from actual completed command events and tool receipts, not from a model-written claim that it used a tool. No explicit model override was passed; these turns used CLI default model selection, and exact backend model snapshots were not asserted.
Delivered evidence
Each live run generated report.html, report.json, junit.xml, pytest-output.txt and the standalone test_agent_regression.py. The test evidence also retains raw events.jsonl, actual tool receipts, prompt, routing snapshot, normalized trace, thread IDs, token usage and a verification manifest with suite/event SHA-256 hashes. The recorded failure was then reproduced locally without an additional model call.
What this establishes
This is real Agent execution against a deliberately introduced application routing defect. It verifies one failure-to-fix integration path. It is not a naturally occurring production incident, proof of a model reasoning defect, or a reliability estimate across repeated runs. No payment or messaging services were called. Live adapters still use the buyer's own runtime, credentials and model allowance; hosted execution and model credits are not included in the $1 one-time skill purchase.
report.html
PDF · document
Example file from a real run - the skill writes it into your workspace.
Connects securely to your tools. The creator never sees your data.
About this skill
Keep a test for the Agent failure you just found
An Agent used the wrong item ID, skipped a lookup, called a forbidden tool or repeated a charge. Save that failure as a small executable regression suite, then test the next version against the same requirements.
What you provide
- A failed tool-call trace and the requirement it violated.
- Expected tool names, argument values, call limits or literal output checks.
- Optional: your local command adapter for running a revised agent.
What you receive
- A normalized JSON suite with explicit, reviewable expectations.
- A standalone parameterized pytest module that can be rerun outside this skill.
- HTML and JSON reports with per-case checks, failures and actual traces.
- JUnit XML and captured pytest output for your existing development workflow.
Tested examples
The package includes an offline broken/fixed adapter demonstration and a normalized public tau-bench historical GPT-4o failure. The latter keeps both a wrong option-ID call and its later correct retry: every relevant call must meet the expected argument contract. Source evidence and the MIT notice are included.
A fresh controlled Codex CLI test also exercised command mode with real model turns: an obsolete storefront SKU route failed 2 of 4 assertions; changing only the route made all 4 pass. The request, prompt, stock tool and suite were unchanged. This is a controlled configuration-defect test, not a production incident or a repeated-run reliability claim.
Clear scope
Recorded-trace replay checks the trace you supply; it does not rerun a model. Command mode executes your adapter and checks its fresh JSON trace. Expectations come from requirements or tool contracts, never from copying a failed response. No semantic LLM judge, hosted runner, automatic repair or general business-state simulator is included. Passing means the listed checks passed.
Python 3.10+ and pytest are required. The runner uses local files and terminal access. Live adapters may require your own API credentials, network access and model budget. The $1 one-time purchase covers the skill, not API usage or a hosted service.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Testing & Debugging.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 4 days ago
- Software
Python 3.10+pytest
Agent with terminal and local file access. Local JSON command adapter for fresh Agent runs. Your own API credentials and model budget for live adapters.
Permissions
Core runner reads the selected JSON suite, creates reports/tests and runs local pytest. It makes no network requests. Command mode executes the supplied adapter without a shell; that adapter may need network/environment access and may have external side effects. Use mocks or an approved sandbox for write-capable tools.