Agent Trace to Regression Tests: Reproduce & Verify

    1

    Turn a failed Agent trace into executable pytest checks. Reproduce tool-call errors, test a revised agent through your local adapter, and keep a standalone test plus HTML and JUnit reports.

    $1

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Agent Trace to Regression Tests: Reproduce & Verify

    Example session with this skill installed

    Use the regression runner with a real Codex Agent and a local stock lookup tool. Reproduce an obsolete storefront SKU mapping, fix only the mapping, then rerun the same request and assertions. Replay the recorded failure too. Keep all model events and tool execution receipts, and report the limits of this controlled test.

    • Read your context and instructions
    • Compiled the agent trace to
    • Generated the document

    Real Agent run: failure reproduced, fix verified

    Executed on 2026-11-12 using Codex CLI 0.166.0 with existing login, Python 3.11.16 and pytest 8.3.3. The existing runner's command mode started two fresh model-backed Agent turns in read-only sandboxes. The inventory tool and stock values were controlled local fixtures.

    RunActual tool callActual Agent answerCheck resultpytest exit
    Real Agent with obsolete routelookup_stock(sku=SKU-24)SKU-24 available=02/4 passed; case FAILED1
    Replay of that recorded failurelookup_stock(sku=SKU-24)SKU-24 available=02/4 passed; case FAILED1
    Real Agent after route correctionlookup_stock(sku=SKU-42)SKU-42 available=174/4 passed; case PASSED0

    Defect and repair

    The storefront contract defined summer-chair as SKU-42 with 17 available units. The obsolete application route mapped it to SKU-24. The Agent read that route, executed the stock tool, and returned the wrong product's stock. The tests caught both the wrong SKU argument and wrong exact output. The repair changed only the route value from SKU-24 to SKU-42. The fixed Agent read the corrected route, executed the same tool and passed all four assertions.

    The four assertions required lookup_stock, limited it to one call, checked every call's sku argument, and checked the exact final response. The prompt, request, tool code, stock data, CLI options and suite expectations were unchanged. Two distinct completed CLI turns and token usage records were retained. Tool calls were normalized from actual completed command events and tool receipts, not from a model-written claim that it used a tool. No explicit model override was passed; these turns used CLI default model selection, and exact backend model snapshots were not asserted.

    Delivered evidence

    Each live run generated report.html, report.json, junit.xml, pytest-output.txt and the standalone test_agent_regression.py. The test evidence also retains raw events.jsonl, actual tool receipts, prompt, routing snapshot, normalized trace, thread IDs, token usage and a verification manifest with suite/event SHA-256 hashes. The recorded failure was then reproduced locally without an additional model call.

    What this establishes

    This is real Agent execution against a deliberately introduced application routing defect. It verifies one failure-to-fix integration path. It is not a naturally occurring production incident, proof of a model reasoning defect, or a reliability estimate across repeated runs. No payment or messaging services were called. Live adapters still use the buyer's own runtime, credentials and model allowance; hosted execution and model credits are not included in the $1 one-time skill purchase.

    report.html

    PDF · document

    Generated

    Example file from a real run - the skill writes it into your workspace.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Keep a test for the Agent failure you just found

    An Agent used the wrong item ID, skipped a lookup, called a forbidden tool or repeated a charge. Save that failure as a small executable regression suite, then test the next version against the same requirements.

    What you provide

    • A failed tool-call trace and the requirement it violated.
    • Expected tool names, argument values, call limits or literal output checks.
    • Optional: your local command adapter for running a revised agent.

    What you receive

    • A normalized JSON suite with explicit, reviewable expectations.
    • A standalone parameterized pytest module that can be rerun outside this skill.
    • HTML and JSON reports with per-case checks, failures and actual traces.
    • JUnit XML and captured pytest output for your existing development workflow.

    Tested examples

    The package includes an offline broken/fixed adapter demonstration and a normalized public tau-bench historical GPT-4o failure. The latter keeps both a wrong option-ID call and its later correct retry: every relevant call must meet the expected argument contract. Source evidence and the MIT notice are included.

    A fresh controlled Codex CLI test also exercised command mode with real model turns: an obsolete storefront SKU route failed 2 of 4 assertions; changing only the route made all 4 pass. The request, prompt, stock tool and suite were unchanged. This is a controlled configuration-defect test, not a production incident or a repeated-run reliability claim.

    Clear scope

    Recorded-trace replay checks the trace you supply; it does not rerun a model. Command mode executes your adapter and checks its fresh JSON trace. Expectations come from requirements or tool contracts, never from copying a failed response. No semantic LLM judge, hosted runner, automatic repair or general business-state simulator is included. Passing means the listed checks passed.

    Python 3.10+ and pytest are required. The runner uses local files and terminal access. Live adapters may require your own API credentials, network access and model budget. The $1 one-time purchase covers the skill, not API usage or a hosted service.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Testing & Debugging.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 4 days ago

    Software
    Python 3.10+pytest

    Agent with terminal and local file access. Local JSON command adapter for fresh Agent runs. Your own API credentials and model budget for live adapters.

    Permissions

    Terminal / Shell
    Read Files
    Write Files

    Core runner reads the selected JSON suite, creates reports/tests and runs local pytest. It makes no network requests. Command mode executes the supplied adapter without a shell; that adapter may need network/environment access and may have external side effects. Use mocks or an approved sandbox for write-capable tools.

    Listed4 days ago

    What's inside

    Frequently Asked Questions

    Replay mode only checks a recorded trace. Command mode runs your supplied local adapter once per case; the adapter receives request JSON on stdin and returns a trace JSON object on stdout. It can call your Agent if you configure it to do so.

    It proves only the explicit tool-use, argument, call-limit and literal-output assertions. It does not infer business success, simulate tool state or provide semantic evaluation. Review the expected behavior before running the suite.

    A one-time purchase of the skill instructions, standalone runner and examples. You need Python and pytest. Hosted execution, API credits, model subscriptions and automatic repairs are not included.

    New in Testing & Debugging