Benchmark Forge

This classifier is small (5 fixed labels) but its own definitions already create fault lines worth probing — most notably that "login problems" is explicitly TECHNICAL while "account access" is explicitly ACCOUNT, and there's no catch-all for out-of-taxonomy tickets. I designed the set around those structural weak points plus the ambiguity/overlap/wording-sensitivity axes you asked about.

Coverage: 5 canonical baseline · 3 contradiction (TECHNICAL vs ACCOUNT boundary) · 4 overlapping-intent · 2 genuinely ambiguous · 2 distractor · 4 perturbation (2 pairs) · 2 contrastive (1 pair) · 2 out-of-taxonomy · 1 stress/compositional · 1 instruction-adherence-under-pressure.The three most diagnostic cases are B1–B3 — they probe a contradiction baked into your own prompt (login failures are TECHNICAL, account access is ACCOUNT, but real tickets blur the two constantly). How the model handles those, plus O1/O4/S1 (explicit action vs. underlying grievance), will tell you more than raw accuracy will.

Also worth noting: N1/N2 aren't really testable in the normal sense — there's no correct label for them under the current 5-category taxonomy, so they're there to quantify how often you'd be force-fitting real traffic rather than to grade the model.

Support Ticket Classifier — Diagnostic Eval Set v1

25 cases targeting: ambiguity handling, category overlap/primary-intent resolution, distractor robustness, phrasing sensitivity, contrastive sensitivity, out-of-taxonomy behavior, and instruction adherence (label-only output) under pressure.

Categories: BILLING · TECHNICAL · ACCOUNT · SHIPPING · CANCELLATION

1. Canonical baseline

2. Contradiction: TECHNICAL ("login problems") vs ACCOUNT ("account access")

The prompt itself puts these two phrases in different buckets even though they describe the same real-world event. This is the single highest-value place to test.

What this pair tests: whether the model resolves the spec's internal ambiguity consistently (B1 and B2 should not both get the same label just because both mention "account") or whether it pattern-matches on the word "account" regardless of framing.

3. Overlapping categories — primary intent resolution

What this tests: consistency of the "primary intent" rule — O1/O4 share a pattern (explicit action + unrelated grievance → action wins) that a good classifier should apply uniformly; O3 tests whether a bug report gets contaminated by its billing context; O2 has no clean answer and should be used to check evaluator/rubric agreement, not scored as strict pass/fail.

4. Genuinely ambiguous

What this tests: since the system prompt forces exactly one label with no "insufficient information" option, these cases reveal whether the model's confidence is calibrated (same vague input classified differently across runs would indicate guessing) rather than whether it picks the "correct" label.

5. Distractors (irrelevant but realistic detail)

6. Perturbation pairs (should NOT flip label — semantic invariance)

What this tests: if P1/P1' or P2/P2' diverge, the model is phrasing-sensitive rather than intent-sensitive.

7. Contrastive pair (SHOULD flip label — semantic sensitivity)

8. Out-of-taxonomy (no category actually fits)

What this tests: this is a taxonomy-completeness gap, not a model failure — worth flagging to you directly since no amount of prompt-tuning fixes it without adding a catch-all category (e.g. OTHER/GENERAL).

9. Stress / compositional

10. Instruction adherence under pressure

Structural findings independent of any single test result

Built-in category collision: "login problems" (TECHNICAL) and "account access" (ACCOUNT) describe overlapping real-world events. Section 2 cases (B1–B3) will reveal whether the model has a consistent implicit rule for this, but the fix is really at the prompt level — consider adding a tiebreaker clause (e.g. "if login/access issues stem from a technical error, use TECHNICAL; if the account itself is locked/restricted with no error, use ACCOUNT").
2.

No fallback category: N1/N2 will always be forced into a wrong label. If off-taxonomy tickets are common in production, add an OTHER/GENERAL bucket or a confidence/abstention mechanism rather than relying on eval results to catch this.
3.

"Primary intent" is undefined: the prompt says to choose the category representing "primary intent" but gives no rule for when an explicit action (cancel/refund) competes with an underlying grievance (bug/shipping/billing). Cases O1–O4 and S1 test whether the model has converged on a consistent implicit rule ("explicit requested action beats background grievance") — if it has, consider codifying that rule explicitly in the prompt so it's not left to inference.