Event-Driven Architecture Designer

Wrote docs/architecture/eda/eda-architecture.md. Complete event-driven architecture specification covering topic topology, delivery guarantees, transactional outbox mechanics, and partition-ordering strategy.

Event-Driven Architecture: Order Fulfillment and Inventory

1. Executive Summary

This architecture establishes an asynchronous event-driven messaging boundary between Order Management System (OMS), Inventory Service, and Billing Service using Apache Kafka. It resolves flash-sale synchronous coupling and stock over-allocation by introducing transactional outbox publishing, partitioned order streams, and idempotent consumer workers.

2. Context and Scope Boundary

2.1 Boundary Definitions

In-Scope: OMS event emission, Kafka cluster topic topology, Inventory allocation consumer, Billing transaction consumer, Dead Letter Queue (DLQ) routing, Schema Registry policies.

Out-of-Scope (Excluded): Customer notification pipeline, data warehouse / BI ETL pipelines, and internal database migrations outside outbox tables.

2.2 System Participants

3. Domain Event Catalog

3.1 Event Classification

OrderPlaced (Domain Event)
├── Producer: OMS
├── Key: order_id
└── Routing: orders.lifecycle.v1
    ├── Subscriber: Inventory Service (reserves allocation)
    └── Subscriber: Billing Service (initiates payment hold)

Note: Order placement is an immutable past fact (domain event), not an RPC command.

4. Channel and Topic Topology

4.1 Topic Layout

4.2 Sizing & Partition Calculation

• Sustained rate: 300 msg/sec; Peak rate: 1,200 msg/sec.
• Target per-partition throughput budget: 100 msg/sec maximum to guarantee latency ≤ 500 ms at p95 under standard single-thread consumer processing (processing budget ~10 ms per record).
• Derived partition count: ceil(1200 / 100) = 12 partitions.

5. Message Contract and Serialization

5.1 CloudEvents Envelope

{
  "specversion": "1.0",
  "id": "e4b2d106-69fc-4b57-9db1-35b86e09e1e2",
  "source": "/services/oms",
  "type": "com.company.orders.placed.v1",
  "datacontenttype": "application/json",
  "dataschema": "http://registry.internal/schemas/orders.placed.v1.json",
  "time": "2026-09-15T10:00:00Z",
  "traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01",
  "data": {
    "order_id": "ord_99281a",
    "customer_id": "cust_4812",
    "items": [
      { "sku": "SKU-BLK-M", "quantity": 1, "price_cents": 4500 }
    ],
    "total_amount_cents": 4500,
    "currency": "USD"
  }
}

6. Ordering and Partitioning Strategy

6.1 Partition Key Assignment

• orders.lifecycle.v1 is keyed by order_id.
• Rationale: Ensures total strict ordering of all state transitions belonging to a specific order (placed -> allocated -> paid).
• Flash Sale Hotspots: Popular products do NOT partition on sku at the ingress topic, preventing Kafka partition skew. Inventory service aggregates SKU stock levels internally using database row locks keyed on SKU.

7. Delivery and Idempotency Architecture

7.1 Producer: Transactional Outbox Pattern

Direct dual-write to PostgreSQL and Kafka is prohibited to prevent dual-write divergence.

1. OMS writes order record and an outbox event into outbox_events table in one local ACID transaction.
2. A relay poller (CDC or scheduled de-queuer) reads unpublished outbox rows, writes to Kafka with acks=all, and marks rows processed.
3. Producer idempotency enabled (enable.idempotence=true, max.in.flight.requests.per.connection=5).

7.2 Consumer: Idempotent Deduplication

• Natural business deduplication via unique constraint in consumer datastore:
  • Table: processed_events (event_id PRIMARY KEY, processed_at TIMESTAMP, consumer_group VARCHAR(64))
• Processing steps:
  1. Consumer reads message containing event_id.
  2. Transaction begins: Check existence of event_id in processed_events.
  3. If exists, bypass processing and acknowledge Kafka offset.
  4. If absent, execute business logic (e.g. stock reservation), insert event_id, commit transaction, and acknowledge offset.

8. Failure, Poison Message, and Replay Strategy

8.1 Error Classification and Handling

8.2 DLQ Replay Mechanism

• Replay from DLQ requires operator execution via administrative script after schema or bug resolution.
• DLQ messages preserve original id, traceparent, and append x-retry-original-topic header.

9. Governance, Evolution, and Schema Registry

• Registry Mode: Confluent Schema Registry with BACKWARD compatibility mode.
• Schema Evolution Rules:
  • Field removal: forbidden unless deprecated across two major versions.
  • Field addition: mandatory default values required for all new fields.
  • Enum expansion: new enum values treated as unknown variant on older consumers.

10. Operational Observability and SLA Verification

10.1 Metrics and Monitoring

• Consumer Lag: Alert triggered if consumer lag on orders.lifecycle.v1 exceeds 5,000 records or consumer lead time exceeds 300 ms.
• End-to-End Latency Metric: Measured via time field in CloudEvents envelope against consumer processing commit timestamp. SLA threshold: p95 <= 500 ms.

Appendix A: Decision Register

Appendix B: Traceability Matrix

Verification

No external automated validator was supplied; manual structural self-check executed:

• Adheres to standard 10-section EDA architectural template.
• All technical limits derived from provided intake metrics (1,200 peak msg/sec, 12 partitions derived via 100 msg/sec per-partition limit).
• Contains no self-granted approvals; recommendations submitted for Architecture Guild ratification.

Next steps

1. Present topic topology and partition plan (DEC-EDA-02) to Core Architecture Guild for formal sign-off.
2. Align Payments and Inventory engineering leads on asynchronous compensation flows in case of payment rejection post-inventory reservation.
3. Validate CDC / Outbox table schema with database administration team to verify WAL generation overhead under 1,200 msg/sec peak write bursts.