See it in action

You say

I'm building a RAG bot for our internal HR docs. I need to make sure it doesn't hallucinate benefits info and always cites the right PDF. Give me an eval suite design.

Your agent does

Here is the eval suite design for your HR RAG bot, written to three files (eval-design-worksheet.md, llm-judge-prompt-template.md, sample-eval-suite.md):

Metrics (each mapped to a failure mode)

Faithfulness (blocking): LLM-as-judge rubric that flags any benefits claim not grounded in the retrieved source.
Citation accuracy (blocking): programmatic check that every cited PDF exists and matches the answer's source.
Correct refusal (warning): checks the bot declines when the docs don't cover the question.

Golden dataset: 10 starter cases spanning common questions, edge cases, an adversarial "invent a policy" prompt, and a correct-refusal case.

Rubrics: anchored 1-to-5 scales with observable descriptions per level.

Judge prompt: mitigates position, length, self-preference, and halo bias, with a human calibration step before you rely on it.

CI gate: fail the build if faithfulness drops below your threshold on any prompt or model change.

Note: this designs and writes the artifacts — you plug them into your own test framework or CI to run them. It does not execute evals or call models.

About this skill

Teams ship LLM features with unit tests for the plumbing and vibes for the model — then a prompt tweak or model upgrade quietly breaks quality and nobody notices until users do. Evals are the missing test suite, and writing them is a craft: bad rubrics measure fluency instead of correctness, and naive judge prompts have known biases. AI Feature Eval Writer does the design and writes the artifacts. Describe your feature, what good looks like, and the failure modes you fear, and it produces the eval plan — each failure mode becomes its own metric with the cheapest grader that works (programmatic checks first, LLM-as-judge only where genuinely needed) and a blocking or warning threshold; a golden dataset design with 8 to 12 concrete starter cases including adversarial and correct-refusal cases; anchored 1-to-5 rubrics with observable level descriptions; ready-to-use judge prompts that mitigate position, length, self-preference, and halo biases, with a human calibration step before you trust them; and the CI regression gate that runs on every prompt or model change. The download includes three reference files: the eval-design worksheet, the LLM-as-judge prompt template with bias guards, and a complete worked sample suite. It designs and writes the artifacts; it does not execute evals or call models. Works with Claude Code, Cursor, Codex CLI, Gemini CLI, and any SKILL.md agent.