Prompt & Eval Dataset Architect

The problem

AI teams can build hundreds or thousands of test cases and still have an eval dataset that proves very little. Easy examples dominate, synthetic rows are mistaken for production reality, near-duplicates leak across splits, labels stay vague, and the failures that matter most never make it into the test set.

What it does

• Designs prompt and eval datasets around the actual system, decision, success criteria, and failure modes being tested.
• Builds hard negatives, ambiguity cases, contradiction tests, counterfactual pairs, perturbation families, and tool-failure scenarios that expose shallow or brittle behavior.
• Separates real production cases, curated examples, synthetic cases, adversarial cases, and regression incidents so dataset provenance remains visible.
• Defines operational labels, scoring rubrics, severity classes, and observable expected behavior without requiring hidden chain-of-thought.
• Audits train, tune, eval, regression, and holdout splits for leakage, near-duplicates, templated shortcuts, and contamination.
• Creates a regression ledger so real AI failures become durable future tests instead of disappearing after a local fix.

Why this beats prompting it yourself

Generic dataset prompts are good at generating more examples. This skill is designed to generate diagnostic examples: cases that can actually prove the system wrong. It prioritizes failure coverage, counterfactual testing, provenance, clean split boundaries, and correction memory instead of treating dataset size as a substitute for evaluation quality.

Use cases

• Build eval datasets for LLMs, RAG systems, coding agents, customer-support bots, classifiers, extraction systems, and tool-using agents.
• Generate hard negatives and boundary cases that distinguish genuine capability from pattern matching.
• Audit existing JSONL or CSV eval sets for leakage, weak labels, duplicate families, and missing high-consequence failures.
• Convert production incidents into regression cases with explicit expected behavior and retest status.
• Create cleaner TRAIN, DEV, EVAL, REGRESSION, and HOLDOUT boundaries before prompt or model iteration.

Known limitations

This skill does not prove that a synthetic dataset represents production, certify statistical sufficiency, guarantee the absence of benchmark contamination, or create reliable gold labels where the domain itself is uncertain. High-quality labels and high-stakes release decisions may still require domain experts, human review, or external evaluation tooling.