Works with the AI Agents you already use
incident derived agent evaluation dataset
Construct provenance-linked, deduplicated evaluation datasets from real-world agent incident logs.
$15
incident derived agent evaluation dataset
Example session with this skill installed
Build an evaluation dataset from the three redacted API timeout incidents in incident-logs/q3-failures.json. Deduplicate and split by incident family.
- Read your context and instructions
- Compiled the incident derived agent
I have processed the incident logs, deduplicated the failure traces, and generated a provenance-linked dataset manifest. The incidents were grouped into the 'Connection Timeout' family and split to preserve a held-out set.
Incident-Derived Dataset Manifest
| Metric | Value |
|---|---|
| Total Incidents Processed | 3 |
| Unique Prompt Families | 1 |
| Deduplicated Records | 2 |
| Source Provenance | incident-logs/q3-failures.json |
| Split Strategy | 50% Production / 50% Held-out |
| Artifact Path | ./outputs/dataset_2026_09_18.json |
Dataset Content Summary
[
{
"id": "CASE-001",
"incident_family": "api_timeout_retry_exhaustion",
"minimal_context": "User requests high-latency reporting endpoint; auth token expires during 30s wait.",
"expected_behavior": "Agent should refresh token or return 408 status; current failure shows 500 loop.",
"provenance_ref": "q3-failures.json:line_45",
"tags": ["network", "auth-lifecycle"]
},
{
"id": "CASE-002",
"incident_family": "api_timeout_retry_exhaustion",
"minimal_context": "Parallel execution of 5 data fetch calls exceeds concurrent connection limit.",
"expected_behavior": "Queue management or exponential backoff implementation.",
"provenance_ref": "q3-failures.json:line_112",
"tags": ["concurrency", "rate-limiting"]
}
]
Next steps
- Run
scripts/build_dataset.pyto export the deterministic JSONL for your evaluation pipeline. - Review the semantic duplicates in
./outputs/review_queue.mdto confirm the deduplication logic. - Apply the held-out set to your CI/CD gate for the next agent release.
Connects securely to your tools. The creator never sees your data.
About this skill
The problem
General LLM benchmarks fail to capture how your specific agent breaks in production. Generic evaluation datasets lack the provenance needed to trace failures back to real-world incident root causes.
What it does
- Extracts minimal reproducible prompts from redacted incident logs and failure traces.
- Groups paraphrases and near-duplicate incidents to prevent data leakage across splits.
- Separates verified facts from proposed explanations to ensure objective ground truth.
- Generates a deterministic dataset manifest with linked source provenance and versioning.
- Identifies label conflicts and semantic duplicates for manual review.
Frameworks & tools
Python scripts (build_dataset.py), JSON schema for incident fixtures, and provenance-linked reporting.
Why this beats prompting it yourself
Manual dataset creation often leads to data contamination where similar prompts appear in both train and test sets. This skill enforces strict evidence discipline, ensuring every evaluation case is linked to a real incident without leaking sensitive credentials or model-generated hallucinations.
Use cases
- Building a gold-standard evaluation set for agentic workflows.
- Deduplicating incident reports to create clean fine-tuning data.
- Auditing agent performance against historical production failures.
- Preparing held-out test sets for rigorous model deployment gates.
Known limitations
Requires pre-redacted incident cases as input. Does not perform model evaluation itself or invent ground truth where source data is missing.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
Requires pre-redacted incident inputs. Does not perform actual model inference or provide accuracy metrics. Coverage is limited to supplied incident families.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 1 month ago
- Software
Python
Permissions
No special permissions declared or detected