LLM App Production Pack

The agent loads llm-cost-control, prices the export and checks the cache columns:

$ node estimate-cost.mjs support-usage.jsonl
source               model              requests  uncached in  cache read  cache write   output  hit rate    cost
support-usage.jsonl  claude-sonnet-5-5     2,000   37,040,000           0            0  879,900      0.0%  $82.88
total $82.88  (prices checked 2026-10-13)

Every request sends about 18,500 input tokens and reads nothing from cache. The prompt-caching reference names the cause: caching is a prefix match, and the system prompt starts with a timestamp and the customer's name, so no two requests share a prefix. The fix freezes the system prompt, adds breakpoints, moves per-request values after the cached prefix, and logs the hit rate so a regression is visible:

export async function reply(history: Anthropic.MessageParam[], userName: string, question: string) {
  const res = await client.messages.create({
    model: "claude-sonnet-5-5",
    max_tokens: 2000,
    tools: TOOLS,
    cache_control: { type: "ephemeral" },
    system: [{ type: "text", text: SUPPORT_POLICY, cache_control: { type: "ephemeral" } }],
    messages: [
      ...history,
      { role: "user", content: `Customer: ${userName}. Date: ${new Date().toISOString().slice(0, 10)}.\n\n${question}` },
    ],
  });
  const u = res.usage;
  const prompt = u.input_tokens + (u.cache_creation_input_tokens ?? 0) + (u.cache_read_input_tokens ?? 0);
  console.log(JSON.stringify({ route: "support", cache_hit: prompt ? (u.cache_read_input_tokens ?? 0) / prompt : 0 }));
  return res;
}

It explains that input_tokens on Claude excludes cached tokens, which is why the hit rate is computed from the sum, and that the 5-minute TTL is right here because support traffic arrives every few seconds. After deploying, the next day's export:

$ node estimate-cost.mjs support-usage-after.jsonl
source                     model              requests  uncached in  cache read  cache write   output  hit rate    cost
support-usage-after.jsonl  claude-sonnet-5-5     2,000    1,280,000  35,044,800      715,200  879,900     94.6%  $16.65
total $16.65  (prices checked 2026-10-13)

Same traffic, 80% cheaper, and cache reads no longer count toward Claude's input-token rate limit. Before suggesting a cheaper model it runs --model claude-haiku-5-5 on the same log ($4.14 at the old token counts) and says the switch needs a pass on the support golden set with run-evals.mjs first. Finally it adds a test from the caching reference that sends the same request twice and asserts the second reads from cache.

The handler compiles with tsc --strict against @anthropic-ai/sdk 0.133.0, and both tables are the estimator's real output on those logs.