LLM Eval Framework Builder

About this skill

You changed the prompt, tried four inputs, it looked better, you shipped — and three days later support tickets say outputs are worse for an entire class of inputs you didn't test. Eval-less LLM development is just deferred debugging with a user-facing blast radius. This skill builds your evaluation framework: the test case set that covers your real distribution (not just the examples you had handy), the scoring rubric that maps to the quality dimensions that actually matter, the automated evaluation pipeline that runs on every prompt change, and the regression detection that tells you when a model upgrade quietly breaks your use case. It also covers the tricky parts — LLM-as-judge calibration, handling non-determinism in pass/fail metrics, and the human evaluation spot-checks that catch what automation misses. Give it your task, your current failure modes, and your quality bar; it returns an eval suite you can run in CI.