Incident Postmortem with Evidence Trail

I've written the draft postmortem: postmortem-2026-09-30-checkout-api.md. It's marked as a draft pending review by the incident owner, and nothing has been posted or sent.

Timeline

Timezone: everything is in UTC. Your Slack times were Pacific Daylight Time on Sept 30 (UTC−7). Two Slack messages land exactly where the alerts say they should, which confirms that offset.

Sources: each timeline row is tagged with the line it came from. ALR is the alerts, CI the deploy log, SLK the Slack export, NOTE your notes.

Response times: about 4 min from deploy to first alert, 1 min to acknowledge, 26–28 min to rollback, 45 min until errors were under 1%, 72 min until it was declared resolved.

When customers were affected: at least 49 min. The true start is unknown; it falls somewhere between the 17:38 deploy and the 17:42 alert.

Places the evidence disagrees (I recorded both times rather than picking one):

• Rollback: the deploy log says 18:12 UTC; Raj's "rolling back now" in Slack is 18:14 UTC.
• max_connections change: the deploy log says 18:20 UTC; Slack says 18:25 UTC.
• Date: you said "yesterday", which would be Oct 1. Every log says Sept 30, so the draft uses Sept 30.
• Day of week: Dan said "Tuesday", but Sept 30, 2026 was a Wednesday.

Keeping it blameless

• I left out Dan's 11:05 "this is on Priya." I kept the question he raised (why it shipped without a canary) and treated it as a process issue: the pipeline lets someone skip the canary by setting a flag by hand.
• The draft names people by role and never names anyone as a cause. A check for "failed to", "should have", "careless", "human error", "fault" and Priya's name came back clean.

Contributing factors

• Retries: card-auth retries had no backoff or cap, so they multiplied load on the database.
• Canary skipped: the canary stage was turned off for this deploy.
• No database alert: nothing alerts when payments-db runs out of connections; only the API errors alerted.

Idle timeout (not confirmed): raising the pool idle timeout probably made connections stay held longer. It's labelled as inferred because no connection-count data supports it yet.

Rollback didn't fully clear it: the database stayed at its connection limit after the rollback, and nothing in the evidence explains why.

Action items

• Your three are in: retry backoff (Raj, marked unconfirmed because your note had a "?"), mandatory canary, and the database connection alert.
• I added two proposed ones, labelled so you can drop them: measure the idle-timeout effect before that change ships again, and find out why the database stayed full after the rollback.
• The P1/P2 priorities are my suggestions; your notes didn't give any. Owners other than Raj are unassigned and there are no due dates, as in your notes.

Gaps. The doc ends with 11 open questions for review. The main ones are:

• Severity, incident owner, and whether anyone acted as incident commander were never stated.
• Why the canary was skipped is unknown; the root-cause analysis stops there.
• Whether max_connections stays at 400. I made no recommendation; that's for normal change review.
• The revenue figure is still pending from finance, and there's no unique-customer count, SLA impact, or status-page record.

No other copies exist.

---

The file it wrote: postmortem-2026-09-30-checkout-api.md

DRAFT - PENDING INCIDENT OWNER REVIEW

Postmortem: checkout-api errors after v4.18.0 deploy (payments-db connection exhaustion)

---

Summary

At 17:38 UTC on 2026-09-30, checkout-api v4.18.0 was deployed. The release raised the DB pool idle timeout and added retries on card-auth calls. Four minutes later, at 17:42, a p95 latency alert fired. By 17:46 the error rate was 34% and the payments on-call rotation was paged. The on-call engineer found that checkout errors were "connection pool exhausted" on payments-db. Failed card-auth calls were each retried 3 times with no backoff or cap, which multiplied the load on the database. Responders rolled back to v4.17.3 (18:12–18:14 UTC; the two sources disagree) and raised payments-db max_connections from 200 to 400 (18:20–18:25 UTC; the two sources disagree). The error rate fell below 1% at 18:31 UTC, and the incident was declared resolved at 18:58 UTC. About 3,100 checkout attempts failed and support received 41 tickets. Revenue impact is still pending from finance.

Contributing factors (summary): retries with no backoff or cap amplified load on a shared DB connection limit (T2). The canary stage was skipped for this deploy through a flag set by hand (P1). There was no alert on DB connection saturation (D1). The raised pool idle timeout is a likely but

unconfirmed contributor (T1, INFERRED).

---

Timeline

Reference timezone: UTC. The Slack export is in Pacific time. On 2026-09-30 that is PDT (UTC−7), checked with Intl.DateTimeFormat for America/Los_Angeles. The offset is supported by two independent matches: the on-call ack at 10:47 PDT = 17:47 UTC, one minute after the 17:46 page, and "error rate under 1%" at 11:31 PDT = 18:31 UTC, which matches the RESOLVED alert. The Slack export has no date, so its date is INFERRED as 2026-09-30 from these matches.

Source key (the evidence as supplied, numbered in the order pasted):

• ALR-1..4: monitoring alerts, lines 1–4 (UTC)
• CI-1..3: deploy log, lines 1–3 (UTC)
• SLK-1..11: #inc-checkout Slack export, messages 1–11 (Pacific)
• NOTE-1..5: incident lead's notes, bullets 1–5

Conflicts

---

Response Metrics

---

Impact

---

Contributing Factors

5 Whys

1. Why did checkouts fail? checkout-api requests failed with "connection pool exhausted" on payments-db. (SLK-2)

Why was the pool exhausted? payments-db was at its 200-connection limit (SLK-8, CI-3). Failed card-auth calls were each retried 3x, which multiplied load (SLK-4). INFERRED: the raised idle timeout in the same release made connections stay held longer (CI-1, T1).
3.

Why did retries make it worse instead of riding through? The retry logic had no backoff and no cap, so retries came immediately and at full volume. (NOTE-2)
4. Why did this reach full production traffic before the effect was seen? The canary stage, which would have exposed the release to a fraction of traffic first, was skipped by a flag set by hand. (NOTE-1)
5.

Why could the canary be skipped? The pipeline accepts a manual skip flag.

The evidence stops here. Why it was set for this deploy, and whether skipping needs approval, is not known (Open Questions #3).

---

What Went Well

Fast detection and response. The first alert fired 4 minutes after the deploy (CI-1 → ALR-1), and the page was acknowledged within 1 minute (ALR-3 → SLK-1).

Fast, accurate diagnosis. Pool exhaustion was identified 2 minutes after ack (SLK-2), and the retry amplification was identified by 18:02 (SLK-4).

External cause ruled out early. The card-auth provider status was checked at 17:55 (SLK-3), so the investigation stayed on internal changes.

• Rollback was available and used. v4.17.3 was restored about 6 minutes after the proposal (SLK-6 → CI-2/SLK-7).

Responders kept checking after the first mitigation. Partial recovery was noticed (SLK-8), and a second mitigation was applied (CI-3/SLK-9).

Recovery was confirmed before closing. The error rate fell below 1% at 18:31 (ALR-4), and the team watched for 27 more minutes before declaring the incident resolved (SLK-11).

What Could Improve

Canary protection was bypassable (P1 → CI-1, NOTE-1). The safeguard existed but could be turned off without, as far as we know, a recorded reason.

No DB-level signal (D1 → SLK-2). Responders had to find connection exhaustion by hand. An alert on payments-db connections would point to the cause directly.

• Retry policy lacked backoff and a cap (T2 → SLK-4). Retries turned a capacity problem into a load amplifier.

Rollback did not fully restore service (T3 → SLK-8). Recovery needed a second change, an emergency max_connections increase. Why connections stayed saturated after rollback is unexplained.

Mid-incident attribution (SLK-5). A "who shipped this" message in the incident channel named an individual while responders were mitigating. Keeping the channel on mitigation, and sending causal questions to the postmortem, keeps responders focused and keeps the review blameless.

Conflicting timestamps between CI and Slack (Conflicts table). Rollback and config-change times differ by 2 and 5 minutes. A single incident log, or bot-posted CI events in the channel, would remove the ambiguity.

---

Action Items

Items 1–3 come from the incident lead's notes (NOTE-5). Items 4–5 are

proposed by this draft to cover factors that otherwise have no action, so accept or drop them in review. All priorities are proposals: the notes gave none.

---

Similar Incidents

None named in the evidence provided. (See Open Questions #9.)

---

Open Questions

Date: the request says "yesterday" (2026-10-01), but every timestamped source says 2026-09-30. Confirm 2026-09-30.
2.

Severity, incident owner, and incident commander: none were stated. Was there a separate incident commander, or did the on-call engineer both coordinate and diagnose?
3.

Canary skip (P1): why was the skip flag set for this deploy? Who can set it, and is a reason or approval recorded anywhere? (The 5 Whys stop here.)
4.

Canary "mandatory" scope (Action 2): should an emergency path remain (e.g., for urgent hotfixes), and if so, what approval should it need?
5.

Rollback and config-change conflicts: can CI job logs or the DB config audit log settle 18:12 vs 18:14 (rollback) and 18:20 vs 18:25 (max_connections)?
6.

max_connections = 400: this was an emergency change during the incident. Is it staying, and has anyone confirmed that payments-db is sized for 400 connections? (This draft makes no recommendation. Any change should go through normal change review.)
7.

Impact gaps: what are the revenue figure (pending from finance), the unique customers affected, the SLO/SLA effect, and the source dashboard behind "~3,100"? Was a status page or customer notice posted?
8.

Impact start time: when did errors actually begin, between 17:38 and 17:42? An error-rate graph would let time-to-detect be measured from impact start.
9. Prior incidents: has checkout-api or payments-db had earlier connection-exhaustion or retry-storm incidents?
10. Responder roles: what is Raj's role (secondary on-call, payments engineer, other)?
11. Action owners and dates: confirm Raj for Action 1, and assign owners and due dates for Actions 2–5.

---

Review and Approval

Next review: 30 days after approval, to check action item completion.