Eval Harness Starter

Every prompt change is a regression risk: you tweak wording, a case that worked yesterday breaks silently, and nobody notices until a user complains. This skill ships a small, dependency-free Python harness (eval_run.py) that runs your prompt against a golden set of known cases, scores each answer with exact-match and rubric rules you define, and writes a Markdown regression report you can diff across runs. It never calls an LLM itself — you plug your own model call into one explicit seam.

Give any agent a repeatable, versioned way to answer "did my prompt change break anything?" The harness owns three jobs: (1) load a golden set in a simple JSONL format, (2) get one answer per case — via your answer function or from recorded predictions — and (3) score every answer and report pass rates plus per-case failures. The agent owns everything the harness doesn't: the prompt, the golden cases, the rubric rules, and the interpretation of a failing run.

• You are iterating on a prompt (system instructions, few-shot examples, output format) and want a stable measurement before/after each change.

• You need to compare two models, two temperatures, or two agent designs on the same task.

• You want a regression suite you can re-run in CI or before shipping a prompt to production.

• You are building a classifier, extractor, router, or any task where "right answer" is checkable by rules.

• Open-ended creative tasks where no rubric you write would be fair (story quality, humor, persuasion) — human review beats a rubric there.

• When you don't yet have 10+ representative cases; build the golden set first, then harness it.

• As a substitute for unit tests on deterministic code — use a real test framework for that.

• If you need statistical significance: a 4-case smoke set shows mechanics, not confidence. 50+ cases is a real eval.