Best AI Model Finder

Workload Selection Plan: Support Classification + Drafts

Decision framing

Workload: classify inbound support email and propose an agent-reviewed draft in English and French.
Volume: 20,000 messages/month (user supplied).
Hard constraints: structured JSON, p95 below 2 seconds, no provider training on content, monthly spend at or below $800.
Unknowns: average input/output tokens, peak concurrency, acceptable draft quality, regions and retention terms. Verify these before selecting a provider.

Candidate shortlist method

Start with current, first-party documentation for models that support the required languages, structured output, latency region, data controls, and availability. Include at least one lower-cost candidate, one quality-focused candidate, and a non-generative baseline for classification if feasible. Record model/version, source URL, check date, price basis, limits, and contract assumptions. Do not infer capability from a generic leaderboard.

Evaluation set

Build a consented, de-identified sample covering both languages, common categories, rare/urgent cases, ambiguous requests, long threads, attachments if in scope, and prompt-injection attempts. Keep a locked holdout set. Human reviewers label the correct category, urgency, required fields, and acceptable response guidance. Exclude unnecessary customer identifiers.

Scoring rubric

Cost model

Estimate monthly cost = 20,000 × (average input tokens × verified input rate + average output tokens × verified output rate) + retries + routing/orchestration + logging + monitoring. Measure token distributions from a representative sample. Compare list and contracted pricing only after current prices are verified. $800 budget implies an all-in average ceiling of $0.04 per message; leave margin for peaks and retries.

Two-week pilot

Days 1–3: define taxonomy, acceptable draft policy, language coverage, privacy requirements, and de-identified test set.
Days 4–6: verify current provider facts from official docs and legal/contract terms; reject any candidate failing a hard constraint.
Days 7–9: run identical workloads across candidates; log versions, settings, latency, retries, and actual cost.
Days 10–11: review errors by language and severity; test prompt injection and malformed outputs.
Days 12–14: compare weighted results and operating complexity; approve a limited human-reviewed pilot only if every hard gate passes.

Exit and monitoring

Pause if p95 exceeds 2 seconds, spend approaches $800, privacy terms are unclear, or critical drafts invent policy. Monitor quality, latency, cost per successful task, drift, and failure rates. Re-test after model/version or prompt changes. No provider or model is recommended until current facts and workload results exist.