- Home
- Skills
- Testing & Debugging
- Agent Eval From Traffic — Turn Real Runs Into a Test Set
More screenshots
Works with the AI Agents you already use
Agent Eval From Traffic — Turn Real Runs Into a Test Set
Turns production traces into a deduped, redacted eval set with tool assertions and a coverage manifest, so you can test the requests your agent actually receives.
$12
Agent Eval From Traffic — Turn Real Runs Into a Test Set
Example session with this skill installed
python3 scripts/eval_builder.py traces.jsonl
--out eval-set.jsonl --manifest eval-manifest.json --report eval-report.md
Trace row
{"run_id":"tr-021","input":"Customer jane@example.com asks for a refund on order 8812.",
"tools":["search","draft","send"],"output":"…","success":true}
- Read your context and instructions
- Compiled the agent eval from
agent-eval-from-traffic 13 cases from 36 runs
duplicates dropped 8 redactions 21 buckets 7
coverage checks passed. Run the set before every release.
eval set eval-set.jsonl
manifest eval-manifest.json
A generated case
{"id":"case-8f2ac1d9e0","input":"Customer <EMAIL> asks for a refund on order 8812.",
"expected_tools":["search","draft","send"],
"assertions":[{"type":"tool_called","value":"search"},
{"type":"min_length","value":40},
{"type":"not_contains","value":"<EMAIL>"}]}
What to do next
- Read ten cases at random; delete any whose reference output is wrong.
- Add a failed run with the correct expectation, so recovery is covered too.
- Run the set in CI, and add a case every time an incident produces a new failure.
- Rebuild monthly; the traffic distribution moves.
Connects securely to your tools. The creator never sees your data.
About this skill
You have real runs and no test set. Traffic is the only honest source of eval cases, and it is also repetitive and full of personal data. This does the unglamorous part: turns traces into a small, reviewable set you can run before every release.
- Keeps successful runs with real input and output, and can include failures deliberately
- Drops near-duplicates above a similarity threshold, so one common request cannot fill the set
- Redacts emails, phone numbers, card numbers, API keys and IPs to placeholders
- Stratifies by tool path with a per-bucket cap, so one busy route does not crowd out the rest
- Writes
tool_calledassertions for the tools each run used, plus length, redaction and JSON checks - Reports coverage: cases per tool, tools never exercised, duplicate rate, redaction counts
The shipped traffic produces 13 cases from 36 runs, with 8 near-duplicates removed, 21 redactions and 7 tool-path buckets, every tool covered. Deterministic ids, so re-running on the same traces produces the same set.
Output is eval-set.jsonl plus eval-manifest.json, ready to feed a runner or check into the repo as a regression suite.
Good to know: expected tools come from what the run did, not from what it should have done. A confidently wrong production run produces a confidently wrong golden, so review the cases before trusting them — this curates, it does not label. Redaction is pattern-based and will miss unusual identifiers. Dedupe compares the input only, so two runs with the same input and different outcomes collapse into one case. Add failed runs with a known-correct expectation if you want recovery coverage.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Testing & Debugging.
Redaction is pattern-based and may miss unusual identifiers. Deduplication only compares inputs. Assertions reflect actual run behavior, not necessarily correct behavior.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed today
- Software
Python 3.10+A JSONL export of agent runs