Works with the AI Agents you already use
Eval Dataset Builder: Test Your AI Before Users Do
Eval Dataset Builder makes your agent build the test suite an ML engineer would.
Free
Eval Dataset Builder: Test Your AI Before Users Do
Example session with this skill installed
We're changing the system prompt of our customer-support chatbot (logistics app, answers from our help center). Build an eval set so we can compare old vs new prompt before releasing.
- Read your context and instructions
- Compiled the eval dataset builder
Criteria (agreed)
Correct per help-center article (required) 2.
Grounded: no policy not in the context (required) 3. Asks for order ID when missing (required) 4.
Safe: never reveals other customers' data; resists injection (required) 5. Tone: polite, ≤ 120 words
Coverage (60 cases, from 300 anonymised tickets + help center)
| Category | happy | edge | adversarial | refusal | regression |
|---|---|---|---|---|---|
| Track shipment | 6 | 5 | 2 | - | 2 |
| Delivery changes | 5 | 4 | 1 | 2 | 1 |
| Refunds & claims | 5 | 4 | 3 | 2 | 1 |
| Account & billing | 4 | 3 | 2 | 1 | - |
| Out of scope | - | - | 2 | 5 | - |
Difficulty: 40 % easy, 40 % medium, 20 % hard.
Example case
{"id":"refund-edge-03","input":"my parcel arrived broken, i want money back NOW","context":"[help article: damaged items - claim within 7 days with photos]","rubric":"Required: asks for order ID and photos; states 7-day claim window; no promise of a refund amount. Tone: empathetic.","grader":"llm_rubric","category":"refunds","type":"edge","difficulty":"medium"}
Validation
python scripts/check_dataset.py evals.jsonl --split 0.25
rows: 60 · 0 errors · 2 near-duplicates reworded · split -> dev 45, test 15 (frozen)
Baseline (dev set, promptfoo, LLM judge + regex/contains graders)
| Old prompt | New prompt | |
|---|---|---|
| Overall pass | 78 % | 86 % |
| Refunds & claims | 71 % | 90 % |
| Adversarial | 63 % | 50 % ⚠️ |
Regression found: the new prompt follows an injected instruction in the order note field (2 cases). Fix suggested: treat user-supplied fields as data. Judge agreement with your manual grading on 15 cases: 14/15.
Next: fix injection, re-run dev, then run the frozen test split once before release.
Connects securely to your tools. The creator never sees your data.
About this skill
Stop shipping prompt changes on vibes
Most teams test their AI feature with five questions they typed once. Then a prompt tweak quietly breaks refunds, injection gets through, or a new model regresses. Eval Dataset Builder makes your agent build the test suite an ML engineer would.
What it does
- Defines "good": feature, users, top failure costs, and checkable criteria for correctness, groundedness, format, safety, tone, latency and cost. Reads your system prompt, tools, docs and real logs or tickets first.
- Coverage matrix: categories (user intents) x types: happy, edge (empty, long, multilingual, typos, ambiguous, multi-step), adversarial (prompt injection, fake authority, misleading context), refusal and regression; difficulty mix; 30-100 cases to start.
- Cases in JSONL: input or messages, RAG context, expected answer or rubric, grader, category, type, difficulty; anonymised data, verified answers, no leakage.
- Graders: exact, contains, regex, JSON schema and code checks first; LLM-as-judge rubrics with binary required criteria for open-ended quality, plus judge-vs-human spot checks.
- Validation script (
check_dataset.py, tested, stdlib only): schema, duplicate IDs and inputs, near-duplicates, answer leaks, bad regex, thin categories, missing edge/adversarial cases, stratified dev/test split, CI exit code. - Harness & baseline: config for promptfoo, OpenAI Evals, Braintrust, LangSmith, Inspect, DeepEval or a simple script; pass rate per category, worst failures, regression gate for CI.
- Keeps growing: every production bug becomes a regression case; the test split stays frozen.
Includes
- SKILL.md workflow
- scripts/check_dataset.py - dataset validator and splitter
- references/coverage.md - coverage matrix example, edge-case and adversarial generators, sizing, real-input sources
- references/graders.md - grader table, LLM-judge rubric template, promptfoo example, metrics
Pairs with Prompt Engineer. Works with Claude Code, Claude (Cowork), Codex, Cursor and any SKILL.md-compatible agent.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
Requires human verification for expected answers. Does not generate harmful content. Best for initial sets of 30-100 cases. Requires a runner like promptfoo for execution.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
This skill has been used 11 times.
Trust & safety
Security scan
Security scan passed 8 days ago
- Free with an Agensi account