API Idempotency Design

Wrote docs/architecture/tasks/pay-refund-001/idempotency-design/idempotency-spec.md. Complete idempotency contract specifying composite key scoping, in-flight concurrency locks, exact cached response replays, and payload conflict detection.

---
skill: idempotency-design
---

# Idempotency Contract: Payment Refund Endpoint [IDEMP-PAY-001]

## Summary

This specification establishes the idempotency contract for `POST /v1/payments/{payment_id}/refunds` in `payment-service v3.1.0` under run ID `pay-refund-001`. It resolves cross-account replay risks by rejecting unscoped global keying in favor of a composite key tuple bound to tenant ID, caller ID, payment resource, and client header. Storage is backed by Amazon DynamoDB with strong read consistency across a 24-hour TTL window, sustaining 350 peak operations/second. The contract specifies deterministic handling for in-flight concurrency (HTTP 409), payload drift on identical keys (HTTP 422), and byte-for-byte replay of finalized responses (HTTP 201).

## Detailed Description

Payment refund operations are side-effecting financial transactions. Network disconnects, client timeouts, and automated consumer retries frequently cause duplicate POST requests. This design introduces a deterministic idempotency interceptor placed at the service entry point before payment business logic execution.

### Criteria and weights

| Criterion | Why it matters here | Weight | Source of the weight |
|---|---|---|---|
| Financial Integrity & Replay Isolation | Duplicate executions risk double refunds and financial ledger discrepancies. | 0.35 | Marcus Vance (Core Payments) |
| Multi-Tenant Key Collision Defense | Shared or un-scoped UUID keys must never allow cross-tenant data access or denial of service. | 0.25 | Sarah Chen (API Guild) |
| Latency & Throughhold Headroom | Interceptor must sustain 350 req/sec without introducing > 15 ms p99 latency overhead. | 0.20 | Workload specification |
| Deterministic Error Semantics | Clients must unequivocally differentiate between in-flight locks, payload conflicts, and successful replays. | 0.20 | Request rules |


### Comparison

| Mechanism Candidate | Storage Backing | Concurrency Handling | Payload Mismatch Defense | Key Scoping Model | Evaluation |
|---|---|---|---|---|---|
| Option A: Unscoped Global Key | Redis Cache | Non-blocking overwrite | Ignored (returns cached data) | Raw client header string | Rejected: Violates multi-tenant isolation; Redis risks failover state loss. |
| Option B: DB Unique Index Lock | Postgres Table | Database row lock | Rejects via constraint | `(tenant_id, key)` | Rejected: High connection pool saturation under 350 peak req/sec bursts. |
| Option C: Composite Scoped DynamoDB (Chosen) | DynamoDB with TTL | Conditional Put (`attribute_not_exists`) | SHA-256 hash comparison (HTTP 422) | `tenant_id#caller_id#payment_id#key` | Selected: Zero connection pool bottlenecks, strong read consistency, bounded 24h retention. |


### Result

Option C is selected. Idempotency records are maintained in DynamoDB with a composite partition key:
`PK = IDEMP#<tenant_id>#<caller_id>#<payment_id>#<Idempotency-Key>`

---

### Required Mechanisms

#### 1. Operation Contract [MC-OC-01]
- **Inputs**: Endpoint `POST /v1/payments/{payment_id}/refunds`, path parameter `payment_id` (UUIDv4), required header `Idempotency-Key` (ASCII string, 16 to 128 characters), request body JSON `{"amount_cents": int, "reason": str, "currency": str}`.
- **Algorithm**: Interceptor validates header presence, extracts authenticated `tenant_id` and `caller_id` from JWT session context, and computes canonical payload fingerprint:
  `payload_hash = SHA256(canonical_json(request_body))`
- **Outputs**: Authorized execution dispatch to refund processing domain service, or synthetic intercepted response (HTTP 201 replay, HTTP 409 in-progress, or HTTP 422 payload conflict).
- **Owner**: Sarah Chen (API Architecture Guild) and Marcus Vance (Core Payments).
- **Failure Handling**: Fail-closed. Missing or malformed `Idempotency-Key` returns HTTP 400 with diagnostic code `ERR_IDEMPOTENCY_KEY_MISSING` before entering domain logic.
- **Verification**: Contract validation test `test_refund_operation_contract_schema()` asserting header validation and parameter parsing.

#### 2. Error Taxonomy [MC-ET-01]
- **Inputs**: Interceptor execution conditions, downstream domain responses, and transient network errors.
- **Algorithm**: Deterministic categorization into four distinct operational outcomes:
  1. `400 Bad Request` (`ERR_IDEMPOTENCY_KEY_INVALID`): Header length < 16 or > 128 chars, or illegal non-ASCII characters.
  2. `409 Conflict` (`ERR_IDEMPOTENCY_IN_PROGRESS`): Concurrent request with identical composite key actively executing. Includes `Retry-After: 2` header.
  3. `422 Unprocessable Entity` (`ERR_IDEMPOTENCY_PAYLOAD_MISMATCH`): Key reused with differing canonical JSON payload hash. Aborts execution immediately.
  4. Downstream Failures:
     - Transient 5xx (e.g. gateway timeout 504, DB unreachable): Lock record deleted; subsequent client retries allowed as fresh attempts.
     - Deterministic 4xx business failure (e.g. 404 Payment Not Found, 400 Already Refunded): Recorded as `status = "RESOLVED"` and replayed identically.
- **Outputs**: Standardized JSON error response matching RFC 9457 Problem Details.
- **Owner**: API Architecture Guild (Error Taxonomy Owner).
- **Failure Handling**: Unhandled internal exceptions default to HTTP 500 without recording `RESOLVED` state.
- **Verification**: Contract test `test_idempotency_error_taxonomy()` covering 400, 409, 422, transient 5xx, and deterministic 4xx cases.

#### 3. Idempotency [MC-ID-01]
- **Inputs**: Incoming composite key `IDEMP#<tenant_id>#<caller_id>#<payment_id>#<Idempotency-Key>`, canonical `payload_hash`, Amazon DynamoDB table `payment_service_idempotency`.
- **Algorithm**:
  1. Interceptor issues conditional put: `attribute_not_exists(pk)` with `status = "IN_PROGRESS"` and `expires_at = now + 86400` (24-hour TTL).
  2. If write succeeds: acquire exclusive processing lease; dispatch call to core payment ledger.
  3. If write fails (`ConditionalCheckFailedException`): execute consistent read (`ConsistentRead=true`).
     - If `status == "IN_PROGRESS"`: return HTTP 409 with `Retry-After: 2`.
     - If `status == "RESOLVED"`: compare incoming `payload_hash` to stored `payload_hash`. If match, return stored `response_status`, stored `response_headers` (with `X-Cache-Lookup: IDEMPOTENT-HIT`), and stored `response_body`. If mismatch, return HTTP 422.
  4. On successful completion of refund: update record atomically to `status = "RESOLVED"` with response payload and headers.
- **Outputs**: Single-execution financial guarantee; deterministic byte replay for identical duplicates.
- **Owner**: Core Payments Lead (Marcus Vance).
- **Failure Handling**: Crashing during processing leaves `IN_PROGRESS` lease with 30-second TTL fail-safe; lease takeover requires verification of ledger status.
- **Verification**: Concurrency stress test `test_concurrent_idempotent_refunds()` simulating 50 duplicate requests per second.

#### 4. Compatibility [MC-CM-01]
- **Inputs**: Existing `payment-service v3.0` API callers, mobile app clients, third-party webhook integrations.
- **Algorithm**:
  - Backward compatibility: Existing client integrations using `Idempotency-Key` continue unaffected; server transparently scopes keys by tenant and caller identity without requiring client header syntax changes.
  - Forward evolution: Any change to the canonicalization algorithm or key scope requires semantic API version bump (`/v2/payments/...`).
- **Outputs**: Zero client disruption for existing tenant callers; OpenAPI 3.1.0 schema specification with header constraints.
- **Owner**: Sarah Chen (API Guild).
- **Failure Handling**: Calls omitting `Idempotency-Key` are rejected cleanly; no silent un-idempotent fallbacks permitted.
- **Verification**: API compatibility diff check `test_idempotency_compatibility_diff()` asserting zero breaking wire changes.

---

### Adversarial Cases and Routing

#### 1. Reject HTTP-Only Specification [ADV-HO-01]
- **Vulnerability**: Assuming HTTP verb semantics (`PUT`, `DELETE`) or simple HTTP header caching alone guarantees idempotency without server-side state machines or distributed locking.
- **Adversarial Mechanism**: Client issues concurrent POST refund calls; an HTTP-only proxy passes both requests to backend payment workers, charging or refunding the account twice.
- **Enforcement & Diagnostic**: Hard requirement for atomic server-side state persistence (DynamoDB conditional writes). Interceptor rejects execution if persistence backend is unconfigured or operating in local non-atomic cache mode, emitting diagnostic `ERR_NON_ATOMIC_IDEMPOTENCY_STORE`.
- **Forbidden Output Behavior**: The system is strictly forbidden from executing financial side effects relying solely on HTTP gateway cache headers without persistent atomic leases.

#### 2. Reject Ambiguous Errors [ADV-AE-01]
- **Vulnerability**: Returning generic HTTP 500 or ambiguous error codes when idempotency locks or payload mismatches occur, causing clients to retry indiscriminately.
- **Adversarial Mechanism**: Client sends modified refund amount under an existing key; server returns generic 500; client retries original payload; server state diverges from client expectations.
- **Enforcement & Diagnostic**: Enforce strict error taxonomy differentiation: HTTP 409 for in-progress operations with explicit `Retry-After`; HTTP 422 for payload drift with diagnostic `ERR_IDEMPOTENCY_PAYLOAD_MISMATCH`. Emitting HTTP 500 for lock contention is forbidden.
- **Forbidden Output Behavior**: System is forbidden from collapsing concurrency locks or payload mismatch rejections into generic 400 or 500 HTTP errors.

#### 3. Reject Breaking Drift [ADV-BD-01]
- **Vulnerability**: Changing key generation semantics, field exclusion rules, or hashing algorithms in a minor release, causing pending or retried requests to miss previous idempotency records.
- **Adversarial Mechanism**: A deployment changes canonical JSON serialization ordering; client retries previous day request; hash mismatch triggers false HTTP 422 or double charge.
- **Enforcement & Diagnostic**: Hashing algorithm and canonicalization rules are version-pinned and protected by contract compatibility test `test_idempotency_compatibility_diff()`. Any modification to canonical schema without major versioning fails CI with diagnostic `ERR_BREAKING_IDEMPOTENCY_DRIFT`.
- **Forbidden Output Behavior**: System is forbidden from altering payload normalization or hash generation logic within the 24-hour retention window of existing recorded keys.

---

### Invariants and Contracts

    Composite Key Isolation [INV-IDEMP-01]
      Idempotency keys must be composite. A key value emitted by tenant A cannot match or collide
      with tenant B, even if the raw Idempotency-Key strings are identical.

    Payload Drift Rejection [INV-IDEMP-02]
      Reusing an Idempotency-Key with altered refund parameters (different amount, currency, or
      destination) must deterministically produce HTTP 422 and execute no ledger mutations.

    In-Flight Mutual Exclusion [INV-IDEMP-03]
      Concurrent requests bearing identical composite keys must never execute business logic
      simultaneously. Exactly one caller gains execution lease; all contenders receive HTTP 409.

## Explicit Unknowns

- DynamoDB item storage cost projections under peak seasonal bursts (G-1).
- Maximum tolerable clock drift between application API containers and AWS NTP time for TTL accuracy (G-2).

## Traceability

| Claim | Classification | Source | Freshness |
|---|---|---|---|
| Peak 350 req/sec, sustained 40 req/sec | provided | Workload intake | Current |
| 24-hour retry retention window | provided | Request constraint | Current |
| DynamoDB storage selection | decided | Marcus Vance & Sarah Chen | 2026-09-15 |
| Redis rejection rationale | provided | Intake specification | Current |
| HTTP 409 for in-progress concurrency | decided | MC-ET-01 | 2026-09-15 |
| HTTP 422 for payload drift | decided | MC-ET-01 | 2026-09-15 |
| Operation schema path & hash | observed | `schemas/payment_refund_v1.json:b8e4f1a2` | 2026-09-15 |
| Contract test suite reference | observed | `tests/contract/test_refund_contract.py:c49a128e` | 2026-09-15 |
| Compatibility diff check | observed | `tests/compat/diff_v30_v31.json:7a3e90bc` | 2026-09-15 |


## Verification

No validator was supplied, so no command was run.

Reviewer self-check against idempotency domain contracts:
- **Collision Protection**: PASS. Partition key binds `tenant_id`, `caller_id`, `payment_id`, and `Idempotency-Key`.
- **Concurrency Protection**: PASS. Atomic conditional put guarantees single-execution lease.
- **Payload Verification**: PASS. Hex SHA-256 canonical hash comparison prevents silent mutation attacks.
- **Storage Lifecycle**: PASS. Native DynamoDB TTL configured for epoch now + 86,400 s.

## Open Decisions

- `DEC-IDEMP-01`: Sarah Chen to determine whether `X-Cache-Lookup: IDEMPOTENT-HIT` header should be exposed in public API gateway documentation or restricted to internal diagnostic logs (Owner: Sarah Chen).

## Next steps

1. Marcus Vance (Core Payments) reviews DynamoDB conditional put write overhead under 350 TPS load test.
2. Implement composite key generator middleware in `services/payment_api/middleware/idempotency.py`.
3. Configure Terraform manifest for `payment_service_idempotency` table with TTL attribute `expires_at`.