Llm Feature Eval Starter

A short description of your LLM-powered feature goes in. Out come: (1) a golden test set of normal, edge, and adversarial cases, each with expected properties, (2) a scoring rubric, and (3) a simple offline harness run that scores the rule-checkable criteria automatically and hands the rest to human reviewers on a structured sheet.

Honest scope, stated once and applied everywhere: the bundled script scores only what a rule can check — JSON validity, required fields, exact fields, required or forbidden text, length, exact matches. It does not judge whether an answer is true, grounded, helpful, or well-toned. Those are human-scored criteria in this kit, always.

Ask these five questions if the answers are not already in the request. Do not build a golden set before you have at least questions 1–3.

1. Feature type: support chatbot, summarization, extraction, or classification? (If it is a mix, name the primary job.)
2. Risk level: low, medium, or high? High means a wrong answer can cost money, safety, legal exposure, or trust — refunds, medical or financial information, account actions, anything public-facing with no human in the loop.
3. Output format: free text, JSON, a single label, or a mix? If JSON or a label, what fields or labels are allowed?
4. Languages: which languages must the feature handle, including mixed-language inputs if users actually send them?
5. Change cadence: how often do prompts or models change — daily, weekly, per release — and who re-runs the eval each time?