Transactional Outbox Design

Wrote docs/architecture/tasks/order-outbox-001/outbox-design/outbox-pattern-contract.md. Complete transactional outbox contract specifying logical replication CDC relay, schema evolution, poison pill DLQ diversion, and partition pruning.

---
skill: outbox-design
---

# Transactional Outbox Contract: Order Management Service [OUTBOX-ORD-001]

## Summary

This specification defines the transactional outbox architecture for `order-service v2.8.0` under run ID `order-outbox-001`, governing the emission of `OrderPlaced` and `OrderCancelled` domain events to Apache Kafka 3.6 from PostgreSQL 15. It resolves the dispute between periodic polling and logical replication by adopting Debezium CDC via PostgreSQL write-ahead log (WAL) decoding, eliminating database connection contention and index table bloat at 1,100 peak writes/second. The design enforces atomic state-and-event commits, guarantees at-least-once Kafka publication within a 500 ms p99 latency SLA, provides a 5-attempt poison message quarantine to `outbox_dlq`, and implements range-partitioned 7-day retention pruning.

## Detailed Description

Dual-writing to PostgreSQL and Apache Kafka in application code suffers from distributed consistency failure when the broker or application crashes between writes. The transactional outbox pattern eliminates dual-write divergence by persisting domain entity updates and outbox events in a single local database transaction.

### Criteria and weights

| Criterion | Why it matters here | Weight | Source of the weight |
|---|---|---|---|
| Database Overhead & Bloat Control | 1,100 writes/sec with aggressive polling induces table bloat, autovacuum lag, and lock contention. | 0.35 | Database Guild (DBA Objection) |
| End-to-End Relay Latency (p99 <= 500 ms) | Order events trigger real-time stock allocation; relay delays cause inventory race conditions. | 0.30 | Intake SLA requirement |
| Zero-Loss Delivery Guarantees | Dropped financial events cause order-fulfillment divergence across downstream accounting services. | 0.20 | Marcus Vance (ARB) |
| Operational Simplicity & DLQ Isolation | Poison messages must not stall consumer groups or block outbox stream progression. | 0.15 | Operational requirement |


### Comparison

| Candidate | Mechanism | DB CPU & IO Impact at 1,100 TPS | Latency SLA (p99) | Failure Blast Radius | As-of |
|---|---|---|---|---|---|
| Option A: Polling Publisher (`SELECT FOR UPDATE SKIP LOCKED`) | App workers poll outbox table every 200 ms, batch update status, and DELETE rows. | High: continuous write-amplification, index bloat, and vacuum overhead. | 450–750 ms (exceeds 500 ms budget under burst load) | Polling workers compete with customer checkout queries. | 2026-09-16 |
| Option B: Debezium CDC via Logical Decoding (Chosen) | Debezium tailing PostgreSQL WAL via `pgoutput` plugin directly to Kafka. | Minimal: sequential WAL reads; zero polling query overhead on primary DB. | 120–280 ms (meets <= 500 ms budget comfortably) | Decoupled process; WAL disk growth if connector disconnects. | 2026-09-16 |


### Result

Option B is selected. Debezium CDC logical replication avoids table churn and query contention on PostgreSQL while achieving p99 propagation latency below 300 ms.

---

### Required Mechanisms

#### 1. Task Contract & Atomic Write Path [MC-WP-01]
- **Execution Boundary**: All order mutations (`orders` table update) and corresponding event records (`outbox_events` table insert) must occur within the same local PostgreSQL ACID transaction block:
  ```sql

```sql
  BEGIN;
  INSERT INTO orders (order_id, customer_id, status, total_cents) VALUES (...);
  INSERT INTO outbox_events (event_id, aggregate_type, aggregate_id, event_type, payload, traceparent) VALUES (...);
  COMMIT;

- **Invariant**: Direct network publishing to Kafka from `order-service` application processes is strictly prohibited.

#### 2. Outbox Table Schema & Partitioning [MC-TS-01]
- **Schema**:
  ```sql
  CREATE TABLE outbox_events (
      event_id UUID NOT NULL,
      aggregate_type VARCHAR(64) NOT NULL,
      aggregate_id VARCHAR(128) NOT NULL,
      event_type VARCHAR(128) NOT NULL,
      payload JSONB NOT NULL,
      traceparent VARCHAR(128) NOT NULL,
      created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
      PRIMARY KEY (created_at, event_id)
  ) PARTITION BY RANGE (created_at);

• Partitioning: Daily table range partitions (outbox_events_y2026m09d16, etc.) eliminating per-row DELETE overhead.

3. Relay Engine & Publishing Semantics [MC-RE-01]

• Relay Component: Debezium PostgreSQL Connector 2.5 with pgoutput plugin.
• Publication Target: Apache Kafka topic orders.lifecycle.v1, partitioned by aggregate_id (order_id) to ensure strict per-order sequence ordering.
• Producer Configuration: acks=all, enable.idempotence=true, max.in.flight.requests.per.connection=5.
• Delivery Guarantee: At-least-once delivery. Downstream consumers must deduplicate using event_id.

4. Error Handling & Poison Message Quarantine [MC-EH-01]

• If serialization fails or destination topic metadata is invalid:
  1. Retry up to 5 times with exponential backoff (50ms, 100ms, 200ms, 400ms, 800ms).
  2. Upon 5th consecutive failure, capture error stack trace and route payload into table outbox_dlq.
  3. Connector advances WAL offset, preventing head-of-line blocking for subsequent valid order events.
• Alerting: Emit P1 alert to Database on-call engineer upon any insert into outbox_dlq.

5. Cleanup & Pruning Lifecycle [MC-CP-01]

• Automated maintenance job drops daily table partitions older than 7 days:

  DROP TABLE IF EXISTS outbox_events_y2026m09d09;
  

• Replaces individual row deletions, preventing table vacuum bloat completely.

Invariants and Contracts

Atomic State and Outbox Binding [INV-OUTBOX-01]
  An order record must never be committed without its corresponding outbox event, and an outbox
  event must never be committed without its order state update.

Prohibition of Dual-Writing [INV-OUTBOX-02]
  Application service containers are denied IAM / network credentials to publish directly to Kafka;
  only the managed Debezium CDC instance holds producer permissions.

WAL Disk Space Safeguard [INV-OUTBOX-03]
  PostgreSQL replication slot `max_slot_wal_keep_size` configured to 20 GB. If Kafka is unavailable,
  WAL accumulation halts at 20 GB to prevent primary database disk exhaustion.

Explicit Unknowns

• Network latency impact of cross-availability-zone Debezium-to-Kafka ingestion during AWS zone degradation (G-1).
• Maximum duration Kafka cluster can remain degraded before hitting the 20 GB WAL safety ceiling at 1,100 orders/sec (G-2).

Traceability

Verification

No validator was supplied, so no command was run.

Reviewer self-check against outbox pattern contracts:

• Atomicity Check: PASS. Schema and write path use single ACID transaction block.
• Bloat Mitigation: PASS. Daily range partitioning with partition dropping replaces DELETE queries.
• Head-of-Line Blocking: PASS. 5-retry DLQ quarantine routes poison messages to outbox_dlq.
• Latency Verification: PASS. WAL streaming delivers p99 latency < 300 ms, satisfying 500 ms SLA.

Open Decisions

• DEC-OUTBOX-01: Database Guild to confirm automated cron script permissions for dropping 7-day-old PostgreSQL partitions (Owner: Marcus Vance).

Next steps

1. Coordinate with Database Guild to configure PostgreSQL logical replication parameters (wal_level=logical, max_replication_slots=5).
2. Deploy Debezium PostgreSQL connector manifest in staging environment and execute 1,100 TPS burst benchmark.
3. Establish monitoring alerts on PostgreSQL WAL replication lag and disk consumption.