AI Eval & Test Suite Quality Gate

AI Eval & Test-Suite Quality Gate

A pre-ship review gate that audits your AI evaluation and test suite for the flaws that quietly make it lie — before you trust a green dashboard to approve a model, prompt, or agent change.

The problem it solves

Evals fail silently. A suite reports 94% and the team ships, but the judge rubric was gameable, the test set leaked into the prompt, the hard cases were never included, and the headline metric measured format instead of correctness. The dashboard turns green while the system gets worse. This gate treats the eval suite itself as the thing under test and tells you whether its passing score means anything.

What it does

It installs a skeptical evaluation reviewer between your eval suite and the decision you are about to make with it. It does not rewrite your evals or invent new test cases. It audits the suite you already have across five passes and returns one structured verdict.

1. Gameable-rubric check — criteria a mediocre answer can satisfy: rewarding length, format, keywords, or confident tone instead of correctness; vague unanchored scales; judge prompts that leak the answer or invite generosity.
2. Data-leakage and contamination check — the test set appearing in the prompt or few-shot block, golden answers visible to the model under test, and calibration/evaluation overlap that turns memorization into a passing score.
3. Coverage and edge-case check — missing failure modes, adversarial and malformed inputs, boundary values, and absent negative tests that are supposed to fail but are never checked.
4. Determinism and statistical-rigor check — non-deterministic scoring with no fixed seed or temperature, single runs reported as stable, thresholds with no sample-size justification, and flaky tests masked by reruns.
5. Metric-validity check — proxy metrics standing in for quality, averages that hide catastrophic tails, and thresholds chosen to clear the current build rather than define acceptable behavior.

What you get

One decision — PASS, REVISE, or FAIL — with each weakness quoted from the suite, tagged by failure class, and ranked by how much it inflates the score, plus the single most likely real-world failure the suite would miss.

Why it works

It separates running an eval from trusting an eval. A model told to assume the suite is flawed and hunt for why a bad system would still pass finds the leakage, gaming, and coverage gaps that a glance at a passing dashboard never surfaces.

What it is not

A reasoning-and-prompting skill, not a test runner, CI system, or coverage tool. It does not execute tests, compute metrics, or connect to your harness — it reads the suite as text and judges its trustworthiness. Pair it with held-out sets, seeded runs, and significance testing for end-to-end rigor.