eval dataset builder

The problem

Shipping AI agents without rigorous testing leads to unpredictable hallucinations and production failures. Manually writing diverse test cases, adversarial prompts, and scoring rubrics is time-consuming and often misses critical edge cases.

What it does

• Generates structured JSONL datasets containing golden examples, edge cases, and adversarial prompts.
• Categorizes test cases by difficulty and type, including core, regression, and multilingual scenarios.
• Constructs comprehensive scoring rubrics with both automated checks and human review criteria.
• Identifies specific failure modes and provides a catalog of latent edge cases for future hardening.
• Provides integration snippets for running evaluations via CLI or Python SDKs.

Frameworks & tools

Outputs compatible with PromptFoo, RAGAS, Claude Code, and standard Python-based LLM evaluation pipelines.

Why this beats prompting it yourself

A simple prompt usually produces repetitive, "happy-path" examples. This skill enforces a specific distribution of difficulty and adversarial traps, ensuring your agent is stress-tested against jailbreaks and boundary conditions rather than just basic queries.

Use cases

• Creating benchmark suites for RAG pipelines to verify retrieval accuracy.
• Generating adversarial datasets to test agent safety and guardrail effectiveness.
• Building golden sets for fine-tuning models or optimizing long-context prompts.
• Establishing regression tests to ensure prompt updates don't break existing features.

Known limitations

Large datasets (50+ examples) may require multiple iterations depending on output window constraints. Requires a clear definition of agent inputs and outputs to be effective.