- Home
- Skills
- Testing & Debugging
- Agent Eval Report Inspector
Agent Eval Report Inspector
Inspect JSON/JSONL agent evaluation reports for pass rates, regressions, and latency metrics.
Free
Works with the AI Agents you already use
Agent Eval Report Inspector
Example session with this skill installed
Inspect the eval report at logs/v2-alpha-run.json and give me a summary of pass rates and any regressions compared to the previous run.
- Read your context and instructions
- Compiled the agent eval report
I have analyzed the evaluation report at logs/v2-alpha-run.json. The inspection identifies a slight dip in pass rates and three specific regression cases in the tool-use category.
Agent Evaluation Summary: v2-alpha-run
| Metric | Value | Status |
|---|---|---|
| Total Cases | 450 | - |
| Pass Rate | 92.4% | ⚠️ Down 1.2% |
| Clean Passes | 398 | - |
| Passes w/ Warnings | 18 | - |
| Flaky Cases | 5 | - |
| Regressions | 3 | ❌ Action Required |
| Timeouts | 2 | - |
| Avg Latency | 1.84s | ✅ Optimized |
Regression Details
case_088_weather_api: Failed (Timeout) - Previously Passedcase_142_sql_injection_filter: Failed (Safety Violation) - Previously Passedcase_301_multi_turn_memory: Failed (Logic Error) - Previously Passed
Latency & Safety
- Max Latency: 12.4s in
case_205_long_context_retrieval. - Safety Outcomes: 48/50 privacy cases passed. 2 failures triggered redaction flags.
Next steps
- Review the
report.jsonllogs forcase_142to inspect the safety filter logs. - Investigate the timeout in
case_088to see if the mock API was unresponsive. - Run the inspector with
--jsonto export these metrics to the release dashboard.
Connects securely to your tools. The creator never sees your data.
About this skill
Analyzing raw agent evaluation data is tedious and prone to human oversight. This skill automates the inspection of JSON and JSONL reports produced by evaluation harnesses to identify pass rates, regressions, and latency issues instantly.
What it does
- Calculate metrics providing pass rates, flaky counts, and total case totals from harness data.
- Identify regressions by flagging cases that moved from pass to fail between runs.
- Track performance using latency statistics to find slow-performing agent steps.
- Surface failures by categorizing timeouts, system errors, and blocking failures.
- Safety reporting by isolating safety and privacy case outcomes for quick review.
How it works
- Load data by reading the standard
report.jsonand its associatedreport.jsonllogs. - Execute command using
scripts/inspect_report.pyfollowed by the target file path. - Toggle output with the
--jsonflag if you need machine-readable summaries for CI/CD pipelines.
Frameworks & tools
This skill works with agent evaluation harnesses that output standard JSON or JSONL formats. It uses a Python-based inspection script to parse attempt logs and metric blocks.
Why this beats prompting it yourself
Parsing nested JSON logs via standard prompting often leads to hallucinated percentages or missed edge cases in large files. This skill uses specific logic to ensure counts are precise and regressions are caught across thousands of lines of logs.
Use cases
- Pre-release audits to verify that a new agent version has no regressions.
- Flakiness detection to find tests that pass and fail inconsistently across attempts.
- Performance monitoring to pinpoint exactly which steps are causing high latency.
- Safety verification to isolate and review all safety-tagged test cases.
Known limitations
Does not execute agents or re-run tests. It only summarizes existing evidence. It cannot infer the root cause of a failure beyond what is recorded in the logs.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Testing & Debugging.
Requires existing JSON/JSONL reports. Does not execute code or follow URLs within reports. Does not provide root cause analysis for failures.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
1 use
Times used to date
Free to use
No account required to browse
Trust & safety
Security scan
Security scan passed 27 days ago
- Free with an Agensi account