End-to-End Deadline and Timeout Design

Wrote docs/architecture/tasks/checkout-timeout-001/timeout-design/timeout-contract.md. Complete deadline and timeout contract specifying end-to-end budget decomposition, per-hop budget subtraction, socket read caps, and cancellation propagation.

---
skill: timeout-design
---

# Timeout & Deadline Contract: Global Retail Checkout Flow [TIMEOUT-PAY-001]

## Summary

This specification establishes the end-to-end deadline budget, layered timeout hierarchy, and distributed cancellation propagation architecture for `checkout-orchestrator v3.0` under run ID `checkout-timeout-001`. It governs synchronous HTTP/gRPC call chains across Cart, Fraud, Payments, and Inventory microservices under a hard 2,500 ms client gateway deadline at 1,100 peak checkouts/second. It decisively resolves the phantom processing failures demonstrated in incident INC-4419 (where downstream services executed payment charges 5,000 ms after mobile clients had already timed out and abandoned transactions). The contract mandates strict deadline propagation via `X-Request-Deadline` headers, dynamic per-hop budget subtraction, immediate fast-fail rejection if remaining budget <= 100 ms, layered socket connect/read bounds, and active cancellation token listeners across all downstream executors.

## Detailed Description

Static, independent per-service timeouts create severe distributed failure modes. When upstream calls consume most of the client's patience, downstream services with uncoordinated static timeouts continue executing expensive mutations long after the customer has disconnected, burning computing resources and creating orphaned database states.

Client Ingress (Hard 2,500 ms Deadline: T_end = now + 2500ms)
│
▼
[ Ingress Gateway ] ──(Injects: X-Request-Deadline: 1789456202500)
│
▼
[ Step 1: Cart Service ] ──────────► Consumes 300 ms (Remaining: 2,200 ms)
│
▼
[ Step 2: Fraud Screening ] ───────► Consumes 400 ms (Remaining: 1,800 ms)
│
▼
[ Step 3: Payment Gateway Call ] ──► Dynamically capped at: min(Static_Max, Remaining - 150ms buffer)
├── Dynamic Timeout: min(1500ms, 1650ms) = 1,500 ms
└── If Client Disconnects ──► Context Cancellation Triggers

### Criteria and weights

| Criterion | Why it matters here | Weight | Source of the weight |
|---|---|---|---|
| Zero Phantom Payment Authorizations | Downstream services must never charge cards for clients that have already disconnected (INC-4419). | 0.40 | Marcus Vance (Lead SRE Architect) |
| End-to-End Latency Bound (p99 <= 2.5s) | Total processing time must never breach the 2,500 ms client experience ceiling. | 0.30 | Consumer Checkout SLA |
| Thread & Connection Pool Protection | Stalled downstream sockets must not hold gateway threads past active client lifetime. | 0.15 | Sarah Chen (Platform Architect) |
| Fast-Fail Short-Circuiting | Abort immediately if remaining time cannot satisfy minimum downstream execution bounds. | 0.15 | SRE Reliability Mandate |


### Comparison

| Timeout Strategy Candidate | End-to-End Deadline Awareness | Phantom Mutation Defense | Dynamic Budget Calculation | Evaluation |
|---|---|---|---|---|
| Option A: Static Per-Hop Timeouts (5s) | None (Each hop gets 5s independently) | Fails: 5 hops can run for 25s total | None (Static values) | Rejected: Caused INC-4419 phantom card charges; thread starvation. |
| Option B: Global Gateway Timeout Only | Edge disconnects at 2,500 ms | Fails: Downstream services keep running | None (Edge-only enforcement) | Rejected: Downstream workers burn DB and CPU capacity on dead requests. |
| Option C: Propagated Deadlines + Subtraction (Chosen) | Complete (`X-Request-Deadline` header) | Enforced: Cancellation tokens halt workers | Calculated at each hop: `T_rem = T_end - now()` | Selected: Strict 2.5s ceiling, zero phantom charges, fast-fail shedding. |


### Result

Option C is selected. An immutable deadline timestamp is injected at the edge and propagated across all internal service boundaries.

---

### Required Mechanisms

#### 1. End-to-End Budget Decomposition [MC-BD-01]
Total Client Deadline: **2,500 ms** (100% budget):

| Execution Phase | Target Component | Allocated Budget | Socket Connect Timeout | Socket Read Timeout |
|---|---|---:|---:|---:|
| Gateway Ingress & Auth | Kong Gateway | 150 ms | — | — |
| Cart State Freeze | `cart-service` | 350 ms | 50 ms | 300 ms |
| Real-Time Fraud Score | `fraud-eval-service` | 500 ms | 50 ms | 450 ms |
| Payment Gateway Authorization | `payment-service` | 1,200 ms | 100 ms | 1,100 ms |
| Inventory Finalization | `inventory-service` | 200 ms | 50 ms | 150 ms |
| Gateway Egress Marshalling | Ingress Reverse Proxy | 100 ms | — | — |


#### 2. Deadline Propagation & Subtraction Formula [MC-DP-01]
- **Header Standard**: `X-Request-Deadline: <epoch_milliseconds_utc>`.
- **Per-Hop Calculation**:
  Upon receipt of request at microservice $N$:
  $$\text{RemainingMs} = \text{HeaderDeadlineMs} - \text{CurrentEpochMs}()$$
- **Short-Circuit Abort Rule**:
  If RemainingMs <= 100 ms:
  - Immediately abort execution.
  - Return **HTTP 504 Gateway Timeout** with header `X-Deadline-Exceeded: true`.
  - Do not invoke downstream database or third-party payment APIs.

#### 3. Layered Socket & Connection Timeouts [MC-ST-01]
- **Connect Timeout**: Capped at 50 ms for internal VPC service mesh calls; 100 ms for third-party payment gateways.
- **Dynamic Read Timeout**:
  $$\text{EffectiveReadTimeout} = \min(\text{ComponentMaxTimeout}, \text{RemainingMs} - 50\text{ ms})$$
  Ensures the local read timeout never exceeds the overall transaction deadline.

#### 4. Cancellation & Phantom Processing Prevention [MC-CP-01]
- All services instantiate Go `context.WithDeadline` or Java `CancellationToken`.
- **HTTP/2 Reset Binding**: If the client disconnects or an upstream service times out, an HTTP/2 `RST_STREAM` frame is propagated downstream immediately.
- Downstream database queries execute using active statement timeouts linked to the remaining deadline:
  `SET LOCAL statement_timeout = '<remaining_ms>ms';`
  Terminates database transactions automatically if the deadline lapses.

---

### Invariants and Contracts

    Mandatory Deadline Header Propagation [INV-TO-01]
      Every internal HTTP and gRPC service call within the checkout flow must forward the
      `X-Request-Deadline` header received from ingress. Stripping or resetting deadlines is prohibited.

    100 ms Fast-Fail Floor [INV-TO-02]
      If calculated remaining time at any hop is <= 100 ms, the service must abort immediately
      with HTTP 504. Schedulers must not dispatch network requests when success is mathematically impossible.

    Cancellation Hook Execution [INV-TO-03]
      Services executing financial card authorizations must bind database connections and HTTP clients
      to the cancellation context. Upon cancellation token trigger, ongoing socket transfers must close.

## Explicit Unknowns

- Clock skew variance across AWS EC2 worker nodes running chrony NTP synchronization (G-1: measured < 2 ms).
- Third-party payment gateway support for cooperative cancellation tokens on initiated authorization requests (G-2).

## Traceability

| Claim | Classification | Source | Freshness |
|---|---|---|---|
| Hard client gateway deadline 2,500 ms | provided | SLA intake constraint | Current |
| Peak 1,100 checkout calls/sec | provided | Traffic profile | Current |
| Incident INC-4419 1,400 phantom charges | provided | Incident review log | Historical |
| 5,000 ms downstream timeout mismatch | provided | Post-mortem evidence | Historical |
| X-Request-Deadline epoch timestamp header | decided | Marcus Vance & Sarah Chen | 2026-09-15 |
| 100 ms fast-fail short-circuit threshold | decided | Architectural invariant INV-TO-02 | 2026-09-15 |


## Verification

No validator was supplied, so no command was run.

Reviewer self-check against timeout architecture contracts:
- **Budget Decomposition**: PASS. Full 2,500 ms budget partitioned across cart, fraud, payment, and inventory.
- **Subtraction Formula**: PASS. Dynamic per-hop calculation bounds downstream read timeouts.
- **Phantom Charge Defense**: PASS. Context cancellation and database `statement_timeout` terminate orphaned work.
- **Fast-Fail Safety**: PASS. Short-circuit floor rejects doomed requests in < 2 ms without socket acquisition.

## Open Decisions

- `DEC-TO-01`: Sarah Chen to determine whether upstream mobile apps should transmit a client-initiated deadline hint or rely strictly on gateway injection (Owner: Sarah Chen).

## Next steps

1. Sarah Chen configures Kong Ingress Gateway plugin to inject `X-Request-Deadline: now + 2500ms`.
2. Platform team deploys deadline subtraction filter in `services/common/middleware/deadline_filter.py`.
3. Conduct staging resilience test injecting 2,300 ms delay in Cart Service to verify zero downstream payment execution.