Production Bug Detective

Production Bug Detective: practical plan

Bug investigation

Define the symptom

Confirm affected users, notification types, time window, app versions, frequency and whether the underlying save is duplicated or only the notification. Avoid collecting message content unless necessary.

Reconstruct evidence

Correlate request IDs, retry attempts, queue messages, database records and notification provider responses. Compare healthy and affected traces. Check recent deployments and provider behavior.

Hypotheses

• Client retries after timeout while server commits successfully.
• Queue delivery is at-least-once without idempotency.
• Consumer acknowledges after sending rather than recording completion.

Reproduction and fix

Create a test that injects a timeout after commit and replays the same request. Add an idempotency key or deduplication record with a bounded retention policy. Ensure notification side effects are transactional or recoverable.

Rollout

Deploy behind a feature flag if possible. Monitor duplicates, dropped notifications and queue lag. Define rollback and data cleanup steps. Document the confirmed cause only after evidence supports it.

Measurement and review

Record the baseline, source, date range, denominator and owner. Use one primary measure plus quality guardrails; compare like-for-like cohorts and disclose missing data. Treat estimates as scenarios, not observed outcomes.

Risks and controls

Check relevance, consent, disclosure, workload and downside before scaling. Keep an evidence trail, obtain human review for material commitments, and pause if trust, quality or compliance signals deteriorate.

Next steps

1. Confirm the goal, audience and available capacity.
2. Gather the minimum evidence for a first test.
3. Assign an owner, budget and review date.
4. Run a small pilot, record outcomes and decide whether to iterate, expand or stop.