Post-Mortem

Real, unedited run on a seeded test incident (a fictional ledger service built for this skill's eval: 11 git commits plus an alert export, chat export, status page and incident report). Graded 15/15 on the skill's checks.

Postmortem: INC-2026-0814, Statement API outage (2026-08-04)

Status: incident resolved. This draft goes to the people involved for correction before it is published.
Tier: Standard. Every account saw real user impact, several factors contributed, and some things are still uncertain. There was no data loss, nothing suggests this is a repeat, and the evidence does not point to a security issue.
Why this incident got a postmortem:

• User-visible impact passed the threshold: about 94% of statement API requests returned 5xx for more than an hour, across all accounts.
• Manual intervention was needed to restore service (a revert).

A human noticed first. Monitoring alerted at 13:58 and again at 14:07. Neither alert reached anyone who acted on it, and responders started work after a customer complaint came in through support at 14:22.

Timezone: all times are UTC.

• Alert export and git timestamps are explicitly UTC (+0000).
• The chat export does not state a timezone. It is treated as UTC because its 14:22 "support pinged us" matches the incident report's 14:22, and the status page is UTC. This is an assumption.

Git dates: author dates. Author and commit dates are the same for every commit in range, so no rebase or cherry-pick skew applies.

Evidence and its limits

Not available:

Anything before 14:25 in chat. Discussion took place in #ledger-eng, which keeps 7 days of history and was not retained [#inc header]. Everything known about 13:58–14:25 comes from the alert export plus statements made afterwards.

No deploy log. Every "this reached production at" claim is bounded by commit times and chat statements, not by deploy records. That covers the 07-26 change, the 15:02 revert and the 18:47 timeout.

No metrics. The incident report says the needed metrics were outside the 14-day retention of the relevant dashboard [incident report]. At today's date (2026-11-02) any 14-day metrics are certainly gone. See open question Q4.

• No logs from the statement API or from db-primary (pg_stat_activity) beyond what the chat paraphrases.
• No account from finance of when the audit export started, how many accounts it covered, or whether it finished.
• No customer-support ticket. The 14:22 complaint is known only second-hand.

No code for the statement API's HTTP handler. The repo has the query layer (ledger/statements.py) and the exporter. It does not show which callers of recent_entries pass a limit.

Timeline

"Believed" lines record what responders understood at the time, kept separate from what the artifacts show was true.

Before the incident: change window

Incident day: 2026-08-04 (Tuesday)

After

The five moments

Contradictions between sources

These are recorded rather than resolved.

1. Impact duration.
   • The incident report's summary says "roughly two and a half hours".
   • Its own impact section, and the status page, say about 14:00–15:18, which is about 78 min.
   • The alert export bounds it at about 68–79 min.
   • Nothing in the evidence supports 2.5 h. One possible explanation is a measurement to the 18:47 timeout change or to the end of the follow-up work, but that is unconfirmed. This document uses the artifact-bounded figure.
2. What the review could see.
   • The incident report says the 500-row limit "had a comment on it explaining exactly why it was there."
   • No revision of ledger/statements.py contains such a comment. The reasoning is only in the 07-14 commit message [git 06452d8].
   • A reviewer looking at the 854b6e9 diff would have seen limit=500 become limit=None, with nothing next to it explaining why 500 mattered.

Pool count after the revert. Alert at 15:04:55 says 11/20. Chat at 15:09 says 14/20. The pool may have gone back up, or the two readings may come from different sources or sampling. This cannot be reconciled from the available evidence.
4.

Time pool exhaustion was confirmed. The incident report says 14:31. The chat shows the pool seen at 20/20 at 14:28 and "confirmed" at 14:31. The machine signal was 14:07. These are consistent if 14:31 means "confirmed by responders". Recorded so 14:31 is not read as the onset.
5.

Metric retention. The incident report, written 2026-08-05, says the metrics needed to count affected users were outside 14-day retention. The outage was one day earlier, so it was well inside a 14-day window. Either the needed metric is a longer baseline, or the retention claim does not fit the dates. See Q4.

Impact

Still outstanding at resolution, and as of HEAD

• The audit export is broken.
  • The revert restored the 500-row cap, which made the full-history export impossible [git 47b71b3].
  • The follow-up pagination change [git 2bbd8e2] calls recent_entries() with an offset argument the function does not accept. At HEAD the export raises TypeError on every call [verified at HEAD].
  • Nothing in the repository after 2bbd8e2 touches statements.py.

No regression test exists for export paging. The commit titled for it [git 50f443f] changes no files. The one existing exporter test [git 256f84c] cannot run as written: it has no import for export_account and no fake_conn fixture. This is consistent with the TypeError above going unnoticed.

statement_timeout: 30s is in place [git 8b09aca]. It replaces a setting that was deliberately unset because "some month-end reconciliation queries legitimately run for several minutes" [git 06452d8]. Three month-ends (Aug 31, Sep 30, Oct 31) have passed since. Whether those reconciliation queries were cut off is not in the evidence (Q6).

Detection and response

What slowed it down:

• Alert routing.
  • The pool alerts go to #ledger-alerts with no pager escalation [alert header].
  • The 5xx alert's route is named api-oncall, yet it also delivered to #ledger-alerts [alert 14:09:30].
  • Neither reached anyone until 14:26:38, after the support escalation. The responders' own assessment at 15:20–15:21 was that this cost about 15 minutes.

"No deploy today." The first hypothesis looked for a same-day deploy. The change had been live for more than a week, so a deploy-focused search could not find it. That is an accurate reading of the deploy history leading to a dead end. Ruling it out took about 2 minutes.

Nothing tied the trigger to engineering. Finance's annual audit run was not visible to responders until they worked out "why today" at 14:53–14:55, after the change was already found.

Slow drain after the revert. In-flight unbounded queries kept holding connections for about 15 minutes after the revert.

What sped it up:

• pg_stat_activity showed the query shape directly (14:44).
• The git history let the responder go from "no LIMIT" to the specific commit in 7 minutes (14:44 → 14:51).
• The batch hypothesis was ruled out in 5 minutes against a concrete timestamp (13:10).
• The revert was clean. There was no migration and no state to undo.

Contributing factors

All six categories were swept. Each factor states whether it is a trigger or a condition.

CF-1: Finance's annual full-history audit export against the largest accounts (trigger)

Evidence: the run started "this morning" on the large reconciliation accounts [#inc 14:55]. ~14 pool connections were running the same unbounded statement query [#inc 14:44]. Those accounts carry "several million entries" [incident report].

• Mechanism: an annual, high-volume caller met an unbounded query for the first time since the default changed.

If this had been different: the same code was live for about 9 days without incident. Without this run, the outage would likely not have happened that day. The latent conditions (CF-2, CF-3, CF-4) would have stayed in place.

Confidence: high that the export was the load, based on the query shape and timing. The run's start time and volume are recollection only.

CF-2: A shared query function's default changed from bounded to unbounded for all callers (condition, from 2026-07-26)

Evidence: limit=500 → limit=None in recent_entries() [git 854b6e9]. The exporter got its unbounded behaviour only through that default [git 3f5e930].

• Mechanism:
  • The need was one caller (the audit export) getting full history.
  • It was met by changing the default of a function other callers share.
  • So any caller that relied on the default became unbounded together, including callers this repository does not show (Q3).

If this had been different: with an explicit opt-in for full history and the default left at 500, the blast radius would have been limited to the audit export's own requests. Whether the statement API's own requests also ran unbounded during the incident is unknown (Q3), so how much this would have narrowed impact cannot be measured.

• Confidence: high for the mechanism. Unknown for its share of the 14 connections.

CF-3: Why the bound existed was recorded only in a commit message (condition / process, from 2026-07-14)

Evidence: the explanation ("keeps this bounded on the large reconciliation accounts") is in [git 06452d8] only. No revision of the code carries a comment, which contradicts the incident report's claim.

Mechanism: the 854b6e9 diff under review showed a number changing, 500 to None. The reason that number mattered was not on screen during review. The reviewer and author saw a change that matched the stated need ("500 rows is not enough").

If this had been different: not quantifiable. It changes what information the review had, not a measurable interval.

Confidence: high for what the diff showed. What the reviewer and author knew from elsewhere is not in the evidence.

CF-4: No time bound on db-primary queries, by design (condition, from 2026-07-14 until 18:47 on the day)

Evidence: statement_timeout "intentionally unset" for month-end reconciliation [git 06452d8]. Changed to 30s at 18:47 [git 8b09aca].

Mechanism: with no row bound (CF-2) and no time bound, a single query on a multi-million-row account could hold a connection for as long as it ran. With 14 such queries in a pool of 20, the statement API could not get connections.

If this had been different: with a 30s timeout in place, each query would have been cancelled after 30s. That would at least have shortened the ~15 min post-revert drain to about 30s. Whether repeated export retries would still have exhausted the pool is a hypothesis that cannot be tested from the evidence.

• Confidence: high for the mechanism. Moderate for the counterfactual.

CF-5: The audit/export workload and the user-facing statement API share one pool of 20 on db-primary (condition)

Evidence: pool: 20 [git 06452d8]. ~14 of 20 connections held by the export query shape [#inc 14:44]. 94% 5xx on the statement API [alert 14:09:30].

Mechanism: a batch-style workload and an interactive one compete for the same small connection budget. Neither has a reserved share.

• If this had been different: not quantifiable from the evidence.
• Confidence: moderate. Pool topology beyond database.yml is not visible.

CF-6: The pool alerts route to an unwatched channel with no paging; the 5xx alert delivers to the same channel despite an "api-oncall" route (detection)

Evidence: route header: "no pager escalation configured" [alert header]. Every CRIT unacknowledged until 14:26:38 [alert]. Human detection via support at 14:22 [#inc 14:25]. "nobody was watching it" [#inc 15:20].

• Mechanism: the alerts fired correctly and on time. The signal reached a channel, not a person.

If this had been different: detection at ~14:07 instead of 14:22 would have removed about

15 of the ~70 minutes of impact (~21%), assuming the rest of the response took the same time. Taking the 13:58 WARN as the trigger to act could have removed more, possibly the whole incident, but that would depend on someone acting before exhaustion, and the evidence cannot support that claim.

• Confidence: high for the 15 minutes. The responders' own estimate agrees [#inc 15:21].

CF-7: A week-old change was outside where the response first looked (response and tooling)

Evidence: "was there a deploy? I don't see one today" → "no deploy today" [#inc 14:29–14:31]. The change was found through pg_stat_activity plus git, not deploy history [#inc 14:44–14:51]. There is no deploy log in the evidence.

Mechanism: the cause was a long-dormant change meeting a new workload. Same-day deploy history, the usual first check, could not reveal it. What found it was reading the live query and tracing it to code.

If this had been different: the deploy check took about 2 minutes, so it cost little directly. The main cost was that cause identification depended on query-level inspection, which began about 14:44.

• Confidence: moderate. The time between 14:38 and 14:44 is unrecorded.

CF-8: Exporter tests do not exercise the exporter (process)

• Evidence:
  • The smoke test [git 256f84c] has no import for export_account and no fake_conn fixture.
  • The "regression test for export paging" commit [git 50f443f] changes no files.
  • At HEAD the exporter raises TypeError [verified at HEAD].
• Mechanism:
  • Before the incident: no test ran the exporter against a large or realistic account, so the unbounded default was never exercised.
  • After the incident: the same gap let a non-working follow-up change land unnoticed.

If this had been different: for 08-04 itself, a test against a multi-million-row fixture would have been needed to surface the problem. The existing suite had no such environment. For the post-incident state, a running test would have surfaced the TypeError at commit time.

• Confidence: high for the state of the tests. Whether CI exists and runs them is unknown.

CF-9: Engineering had no visibility into finance's annual audit run (organizational)

• Evidence:
  • The responders learned of the run during the incident, at 14:55 [#inc 14:55].
  • The 07-26 change was made for that run, yet its timing on 08-04 was not known to the people on call. The pre-14:25 chat is lost, so this is not fully confirmed.
• Mechanism: a once-a-year, high-load workload ran with no signal to the team operating the system it loaded.

If this had been different: not quantifiable. It could have changed when "why today" was answered, which was 4 minutes after the cause was found.

• Confidence: moderate. Rests on chat statements, and earlier context was not retained.

Swept with nothing found:

• Trigger: no deploy or config change on 08-04 before the incident [#inc 14:31], [git].

Organizational: no evidence of a prior incident, an earlier postmortem, or a risk that was raised and deprioritized.

Detection: no evidence that any alert misfired or was mis-thresholded. The thresholds fired correctly. The gap is entirely in routing (CF-6).

How it fits together: the audit run triggered it (CF-1). It was harmful because the shared query had become unbounded (CF-2, with CF-3 shaping how that change was reviewed), nothing bounded query time (CF-4), and the export and the API shared a small pool (CF-5). It lasted ~70 minutes partly because the alerts reached no one (CF-6, ~15 min), and partly because of how long it took to identify a dormant change and drain the pool (CF-7, CF-4). The test gap (CF-8) applies both before and after the incident.

What went well, and where we got lucky

What worked, and is load-bearing:

The pool alerts are correctly thresholded. The 18/20 WARN fired 8.5 minutes before exhaustion. The signal existed. CF-6 is about where it went, not whether it fired.

pg_stat_activity showed the query shape directly. It is what turned "pool exhausted" into a specific query within one inspection.

The git history was clean and descriptive. The 07-14 commit message recorded why the limit existed. The 07-26 message recorded the change in default. Together they let a responder go from symptom to commit in 7 minutes.

• The revert was clean. No migration and no forward-only state, so mitigation needed one commit.

The revert's side effect was communicated. The revert commit records that it re-broke the audit export and that finance were told.

A class-level change was made the same day. statement_timeout at 18:47 addresses every unbounded query, not just this one.

Where we got lucky:

It happened at ~14:00 UTC on a Tuesday, during business hours, with support staffed and customers active enough to complain within about 15 minutes. With the pool alerts unpaged (CF-6), an overnight or weekend start had

no detection path except a customer. Impact would have lasted until someone checked a channel or a complaint reached a human.

The trigger was a once-a-year job. The unbounded default sat in production for about 9 days. If the statement API's own callers had relied on the default with large accounts (Q3), it could have broken sooner, under less understood conditions.

The failure was loud, not wrong. Pool exhaustion produced errors, not truncated or partial statements. A row cap misapplied the other way, for example silently truncating the audit export to 500 rows, would have produced wrong data with no alert at all.

Both responders were online and knew the subsystem. One of them traced the change to its commit from memory of the code ("went optional on the 26th").

• Revert was possible because the exporter was the only consumer needing the new behaviour. Reverting cost an internal workflow (finance), not a customer-facing one.

What was actually done

Mitigation, 15:02 → 15:17. 854b6e9 (and the exporter comment) reverted, restoring the 500-row default [git 47b71b3]. This was a workaround in the sense that it removed the new capability rather than delivering it safely. The pool drained and 5xx returned to baseline at 15:17:20 [alert].

Class-level guard, 18:47. statement_timeout: 30s set on production [git 8b09aca], overriding the earlier decision to leave it unset for month-end reconciliation.

Attempted fix for the audit export, 2026-08-05 11:18. Export rewritten to page through results 5,000 rows at a time [git 2bbd8e2].

As committed, it does not work. It passes an offset keyword that recent_entries() does not accept, and raises TypeError at HEAD [verified at HEAD].

• A commit titled "add a regression test for export paging" landed 2026-08-06 with no file changes [git 50f443f].
• Status page updated at 14:40, 15:10 and 15:25 [status].
• Incident report written 2026-08-05 [incident report].

Left in place / loose ends:

The audit export is non-functional at HEAD. Finance's full-history export has not worked since 15:02 on 2026-08-04, as far as this repository shows. Whether finance completed the annual audit by other means is unknown.

statement_timeout: 30s overrides a documented need for multi-minute month-end reconciliation queries. Its effect on the three month-ends since is unknown.

• The pool alert route is unchanged as far as the evidence shows. Nothing in the documents or the repository records a change to the db-primary-pool routing or to the api-oncall delivery target. Alert configuration does not live in this repo, so this is "no record of a change", not "confirmed unchanged".
• The exporter test suite still cannot run the exporter.

About the incident report written on 2026-08-05

That document is cited above as evidence of what people believed the next day. Where it differs from the artifacts, this record follows the artifacts:

• Duration. It says "two and a half hours". The artifacts support about 70 min.

The comment. It says the 500-row limit "had a comment on it". No revision of the code does. The explanation was in a commit message.

The alerts. Its timeline begins at 14:22 and leaves out the 13:58 WARN and the 14:07 CRIT, which is the detection finding.

The cause. It names one cause and attributes it to an individual. This record finds nine contributing factors, most of them conditions that existed before the 07-26 change and remained after the revert.

• "What we must do". Its list of actions is outside the scope of this record and has not been evaluated here.

Open questions

These could not be settled from the evidence and are marked open:

• Q1. When did 854b6e9 reach production? There is no deploy log. "Over a week" is a chat statement.
• Q2. When did the audit export start, and why did impact begin around 14:00 if the run started "this morning"? Possible explanations include a gradual build-up of long-running queries, the order in which the export reached the largest accounts, or a start time later than "morning". There is no evidence to choose between them.
• Q3. Which callers of recent_entries() relied on the default between 07-26 and 08-04? In particular: were the statement API's own requests unbounded during the incident, or was the pool consumed only by the export? The ~14 connections are described as "the same statement query", which does not tell the two callers apart.

Q4. How many users or requests were affected? The incident report gives 14-day retention as the reason no figure exists. That does not fit a report written one day after the outage. Whether a usable figure ever existed is unclear, and any 14-day data is gone now.

• Q5. Were other services on db-primary affected?
• Q6. Have month-end reconciliation queries run into the new 30s statement_timeout on 08-31, 09-30 or 10-31?
• Q7. Why did the 5xx alert on route api-oncall deliver to #ledger-alerts? Is that the route's intended target, or a misconfiguration?

Q8. Pool count after the revert: 11/20 at 15:04:55 per the alert, 14/20 at 15:09 per chat. Which is right, and did the pool rise again?

Q9. Is there another branch, repository or deploy where the offset parameter exists and the paginated export works? Only main exists in this repository.

• Q10. Did finance's annual audit complete, and was it delayed?

Blamelessness pass: done on this final text. Changes are attributed to commits, not people. Names appear only on actions, never on errors. The draft was scanned for counterfactual phrasing ("should have", "failed to", "didn't notice"), hindsight markers and prescriptive language. Counterfactual figures are stated as measurements of what happened, and this document makes no recommendations.

Postmortem for INC-2026-0814 is ready:

• Impact: about 70 min of user impact; about 94% of statement API requests failed, across all accounts.

Contributing factors: 9, across all 6 categories (trigger, latent condition, detection, response and tooling, process, organizational).

• Open questions: 10 that the evidence could not settle.

Sources: git 06452d8..50f443f (2026-07-14 → 2026-08-06, 11 commits, the full history), plus 4 incident documents (incident report, alert export, chat export, status page).

• Blamelessness pass: complete, including the prescription check.

Two things in the "Still outstanding" section are worth bringing to the system owner's attention when this circulates:

• The audit export raises TypeError at HEAD.
• The "regression test" commit is empty.

Neither came up in the incident documents.

Next steps

1. Circulate to the responders and to the author and reviewer of 854b6e9 for correction first. In particular, ask them to correct the pre-14:25 period, where the chat history is lost.
2. Then circulate more widely.

What to do about any of it is a decision for whoever owns the ledger service.