Failmap: know how your AI feature fails, then guard it

When an AI feature "feels off", nobody can say how often it fails or how. Teams reach for a generic helpfulness score, one judge prompt that grades everything and was never checked against people, and prompt changes that break something nobody notices. Failmap replaces that with named failure modes found in your real traces, counted, checked automatically and guarded on every change.

What it does

• Real traces first: inputs, outputs, tool calls and metadata exported to a local JSONL outside the repo, personal data stripped. No users yet: 20 to 50 realistic inputs.
• Diverse sample: clusters on cheap features and adds about a third random picks, 20 to 30 records for the first pass.
• Open coding: a plain-words note on the first thing that went wrong in each record.
• Failure taxonomy: named modes with a definition, a count and two example record ids.
• Code checks first: schema, regex, required fields, citations, tool arguments, output language, each tested on pass, fail and borderline cases.
• One binary judge per mode: only where code cannot decide, with Pass and Fail definitions, labelled examples and a pinned dated model.
• Judge validation: true positive and true negative rates on a held-out split; aim for both above 90%, below 80% is not ready.
• Golden set and CI gate: one eval command that fails the run when a mode drops below its baseline, with cost bounded by caching and a per-run budget.

How it works

1. Collect, sample and open-code traces.
2. Group into modes and re-read earlier records.
3. Fix prompt gaps before building any evaluator.
4. Write code checks, then judges, and validate each judge.
5. Freeze the golden set and wire the eval command into CI.

Frameworks & tools

Any tracer or log that can export traces. The gate runs as npm run eval or pytest evals/ in CI.

Why this beats prompting it yourself

Asking a model to "evaluate my chatbot" produces a Likert score and a generic checklist. This process starts from your own failures, keeps judge examples out of the test split and refuses to trust a judge that has no measured agreement with people.

Use cases

• Chat, extraction or classifier features with no failure rates.
• Before a prompt, model or retrieval change.
• RAG and tool-using agents, with retrieval or per-step modes.

Known limitations

Not for match scoring with have, partial and missing labels, or for uptime, error alerts and spend caps. Reading traces and labelling by hand is required; the skill does not skip that step.