More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Agent Eval From Traffic — Turn Real Runs Into a Test Set

    1

    Turns production traces into a deduped, redacted eval set with tool assertions and a coverage manifest, so you can test the requests your agent actually receives.

    $12

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Agent Eval From Traffic — Turn Real Runs Into a Test Set

    Agent Eval From Traffic — Turn Real Runs Into a Test Set

    Example session with this skill installed

    python3 scripts/eval_builder.py traces.jsonl
    --out eval-set.jsonl --manifest eval-manifest.json --report eval-report.md

    Trace row
    {"run_id":"tr-021","input":"Customer jane@example.com asks for a refund on order 8812.",
    "tools":["search","draft","send"],"output":"…","success":true}

    • Read your context and instructions
    • Compiled the agent eval from

    agent-eval-from-traffic 13 cases from 36 runs
    duplicates dropped 8 redactions 21 buckets 7

    coverage checks passed. Run the set before every release.

    eval set eval-set.jsonl
    manifest eval-manifest.json

    A generated case
    {"id":"case-8f2ac1d9e0","input":"Customer <EMAIL> asks for a refund on order 8812.",

     "expected_tools":["search","draft","send"],
     "assertions":[{"type":"tool_called","value":"search"},
                   {"type":"min_length","value":40},
                   {"type":"not_contains","value":"<EMAIL>"}]}
    

    What to do next

    1. Read ten cases at random; delete any whose reference output is wrong.
    2. Add a failed run with the correct expectation, so recovery is covered too.
    3. Run the set in CI, and add a case every time an incident produces a new failure.
    4. Rebuild monthly; the traffic distribution moves.
    manifest eval-manifest.json

    Connects securely to your tools. The creator never sees your data.

    About this skill

    You have real runs and no test set. Traffic is the only honest source of eval cases, and it is also repetitive and full of personal data. This does the unglamorous part: turns traces into a small, reviewable set you can run before every release.

    • Keeps successful runs with real input and output, and can include failures deliberately
    • Drops near-duplicates above a similarity threshold, so one common request cannot fill the set
    • Redacts emails, phone numbers, card numbers, API keys and IPs to placeholders
    • Stratifies by tool path with a per-bucket cap, so one busy route does not crowd out the rest
    • Writes tool_called assertions for the tools each run used, plus length, redaction and JSON checks
    • Reports coverage: cases per tool, tools never exercised, duplicate rate, redaction counts

    The shipped traffic produces 13 cases from 36 runs, with 8 near-duplicates removed, 21 redactions and 7 tool-path buckets, every tool covered. Deterministic ids, so re-running on the same traces produces the same set.

    Output is eval-set.jsonl plus eval-manifest.json, ready to feed a runner or check into the repo as a regression suite.

    Good to know: expected tools come from what the run did, not from what it should have done. A confidently wrong production run produces a confidently wrong golden, so review the cases before trusting them — this curates, it does not label. Redaction is pattern-based and will miss unusual identifiers. Dedupe compares the input only, so two runs with the same input and different outcomes collapse into one case. Add failed runs with a known-correct expectation if you want recovery coverage.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Testing & Debugging.

    Redaction is pattern-based and may miss unusual identifiers. Deduplication only compares inputs. Assertions reflect actual run behavior, not necessarily correct behavior.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed today

    Software
    Python 3.10+A JSONL export of agent runs

    Permissions

    Terminal / Shell
    Read Files
    Write Files
    Listedtoday

    What's inside

    Frequently Asked Questions

    This skill takes raw agent logs (traces) and cleans them for testing by removing repetitive data, masking sensitive PII, and generating automated validation checks based on how your agent interacted with its tools.

    It is compatible with any AI agent that can export its execution history—including inputs, tool calls, and outputs—into a standard JSONL format. Since it uses Python's standard library, it works across different agent architectures without requiring specific model provider keys or external dependencies.

    No, the skill uses deterministic, pattern-based redaction and logic-based deduplication locally on your machine. This ensures that your production data is never sent to an external LLM provider during the cleaning and evaluation set generation process.

    You receive the core Python builder script, a configuration guide for setting stratification thresholds, and a template for the 'eval-report.md' generated after each run. This includes everything needed to integrate the tool into your existing data engineering or CI/CD pipelines.

    The skill uses pattern-based redaction to mask standard identifiers like emails, API keys, and credit cards. While highly effective for structured PII, we recommend a brief manual review of the generated set if your logs contain highly non-standard or industry-specific sensitive identifiers.

    The generated assertions (like 'tool_called' or 'json_valid') are based on the agent's actual behavior in the trace. This makes it a powerful tool for regression testing to ensure future versions of your agent maintain the same successful logic paths as the original production run.

    New in Testing & Debugging