Incident Postmortem Writer

Postmortem: Checkout Service Degradation (HTTP 500)

Severity: SEV-1 Duration: ~21 min (14:02–14:23)

Impact Summary

Checkout returned HTTP 500 for most users for roughly 21 minutes following a configuration deploy. Customers could not complete purchases until the change was rolled back and error rates returned to normal at 14:23.

Timeline

• 14:02 — Configuration change deployed to production.
• 14:05 — First 500s observed; PagerDuty pages on-call. (At this point responders did not yet know the deploy was the trigger.)
• 14:11 — Responders raise the recent config PR as a possible cause.
• 14:20 — Change rolled back.
• 14:23 — Error rates return to baseline; incident resolved.

Contributing Factors

• The config deploy pipeline had no automated validation to catch the faulty change before it reached production.
• The 5xx alert threshold (5% error rate) delayed detection, so paging lagged the first user-facing errors.

What Went Well

• Once the config PR was suspected, rollback was fast and fully resolved the issue.

Action Items

• Add a config-validation gate to the deploy pipeline. (Owner: TODO — assign) (Due: TODO) — addresses Factor 1
• Lower the 5xx alert threshold / add rate-of-change alerting. (Owner: TODO — assign) (Due: TODO) — addresses Factor 2

Lessons for Others

• A fast, well-understood rollback path limits blast radius more than any single prevention control.

TODO: Follow-up owners and due dates were not provided in the source notes — assign before publishing.