Retry Attempt and Budget Design

Wrote docs/architecture/tasks/sms-notify-retry-001/retry-design/retry-strategy-spec.md. Complete retry strategy specification establishing decorrelated full-jitter backoff, error classification matrices, a 10% global retry budget cap, and SMS provider anti-stampede protections.

---
skill: retry-design
---

# Retry Strategy Specification: Two-Factor SMS Gateway [RETRY-SMS-001]

## Summary

This specification establishes the retry policy, backoff mechanics, and retry budget architecture for `auth-sms-gateway v2.0` under run ID `sms-notify-retry-001`, governing remote SMS dispatch via external Telephony Provider API (`POST /v1/messages`) across 1,200 peak requests/second. It resolves the catastrophic self-inflicted retry storm demonstrated in incident INC-2901 (where immediate static retries amplified traffic to 9,600 req/sec and triggered a 35-minute vendor IP ban). The contract strictly outlaws immediate un-jittered retries, enforcing a hard 3-attempt ceiling, full-jitter exponential backoff (base 100 ms, ceiling 3,000 ms), a client-side token-bucket retry budget restricting retries to <= 10% of outbound volume, deterministic error classification, failure containment, and safe state transitions across telephony outages.

## Detailed Description

Uncoordinated client retries amplify downstream outages into complete service collapses. When a downstream SaaS dependency experiences transient rate-limiting (HTTP 429) or brownouts, synchronized client retries create harmonic waves of requests ("thundering herd"), converting minor hiccups into sustained vendor security lockouts.

Client Auth Request (1,200 req/sec)
│
▼
[ Retry Budget Evaluator (Token Bucket: Max 10% Overhead) ]
├── Budget Exhausted (> 10% retries active) ──► Fast-Fail HTTP 503 (No Retry Allowed)
└── Budget Available
│
▼
[ Attempt 1: Invoke Telephony Provider API ]
├── HTTP 201 Created ──────────► Success; deposit token into retry bucket
├── HTTP 400/401/404/422 ──────► Permanent Error: Terminate Turn (Zero Retries)
└── HTTP 429 / 503 / SocketDrop ─► Transient Error: Enter Full Jitter Backoff
│
▼
[ Backoff Scheduler: Sleep(random(0, min(3000ms, base * 2^attempt))) ]
├── Attempt 2 (Backoff: 0–200 ms)
└── Attempt 3 (Backoff: 0–400 ms) ──► Exhausted ──► Route to Fallback Queue

### Criteria and weights

| Criterion | Why it matters here | Weight | Source of the weight |
|---|---|---|---|
| Vendor Stampede Prevention (Zero IP Bans) | Multiplying traffic during outages triggers third-party IP blacklisting (INC-2901). | 0.40 | Marcus Vance (Security Architect) |
| Latency SLA Compliance (p95 <= 1,200 ms) | 2FA verification SMS must land within user login attention windows. | 0.25 | Intake SLA requirement |
| Non-Transient Error Rejection | Retrying bad phone numbers or schema errors burns money and exhausts quotas uselessly. | 0.20 | Sarah Chen (Telephony Lead) |
| Global Capacity Headroom | Retries must never consume more than 10% of total system outbound network bandwidth. | 0.15 | Infrastructure Platform Policy |


### Comparison

| Candidate Strategy | Backoff Algorithm | Maximum Attempts | Retry Budget Cap | Thundering Herd Risk |
|---|---|---|---|---|
| Option A: Static Immediate Retry (Dev Proposal) | 500 ms constant sleep | 5 attempts | None (100% amplification) | Catastrophic: 8x multiplier triggered INC-2901 ban. |
| Option B: Exponential Backoff without Jitter | 100 ms * 2^attempt | 3 attempts | None | High: Synchronized retry waves still collide at discrete intervals. |
| Option C: Decorrelated Full-Jitter + Budget (Chosen) | random(0, min(3000, 100 * 2^attempt)) | 3 attempts | Hard 10% Token Bucket | Minimal: Smooth traffic distribution, mathematically bounded. |


### Result

Option C is selected. Full-jitter exponential backoff flattens traffic spikes, and the 10% token budget prevents retry storms from compounding downstream stress.

---

### Required Mechanisms

#### 1. Failure Mode [MC-FM-01]
- **Inputs**: Upstream 2FA trigger events, downstream HTTP status codes (`429 Too Many Requests`, `503 Service Unavailable`, `504 Gateway Timeout`), TCP socket resets (`ECONNRESET`), socket read timeouts.
- **Algorithm**: Inspect response status and headers. If transient (429, 503, connection drops), check retry budget and idempotency guard; if permitted, schedule retry under backoff. If permanent (400, 401, 403, 404, 422), terminate immediately without retry.
- **Outputs**: Classified failure record tuple `(CLASSIFICATION, HTTP_STATUS, RETRY_ELIGIBLE, CONSUMED_BUDGET)`.
- **Owner**: Marcus Vance (Security Infrastructure Architect).
- **Failure Handling**: If the failure classification engine encounters unrecognized status codes or unparseable headers, treat as non-retryable terminal failure to avoid uncontrolled retry loops.
- **Verification**: Chaos fault injection test `tests/fault_injection/test_sms_failure_mode_matrix.py` (Freshness: 2026-09-15) asserting exact eligibility routing across 14 failure injection patterns.

#### 2. Policy [MC-PO-01]
- **Inputs**: Attempt counter (1-indexed), initial timestamp, cumulative elapsed time, downstream `Retry-After` header.
- **Algorithm**:
  - Maximum attempts: 3 (1 initial dispatch + max 2 retries).
  - End-to-end operation deadline: 4,000 ms.
  - Exponential backoff with AWS Full Jitter:
    $$T_{\text{sleep}} = \text{random\_between}\left(0, \min(T_{\text{cap}}, T_{\text{base}} \times 2^{\text{attempt}})\right)$$
    where $T_{\text{base}} = 100\text{ ms}$, $T_{\text{cap}} = 3,000\text{ ms}$, multiplier = 2.0.
  - If downstream returns valid `Retry-After: N` (where $N \le 3$ seconds), set $T_{\text{sleep}} = \max(T_{\text{sleep}}, N \times 1000\text{ ms})$. If $N > 3$ seconds, fail fast without retry.
  - Global retry budget token bucket: deposit 0.1 tokens per primary success, withdraw 1.0 token per retry. Retries blocked when balance <= 0.
- **Outputs**: Scheduling delay $T_{\text{sleep}}$ or immediate termination signal.
- **Owner**: Sarah Chen (Telephony Reliability Lead).
- **Failure Handling**: If calculated $T_{\text{sleep}}$ exceeds remaining operation deadline budget, abort retry immediately and emit terminal deadline exceeded error.
- **Verification**: Backoff distribution verification `tests/unit/test_jitter_and_budget_policy.py` (Freshness: 2026-09-15) asserting delay bounds and <= 10% traffic ceiling over 10,000 simulated runs.

#### 3. State Transition [MC-ST-01]
- **Inputs**: Outbound dispatch lifecycle triggers (`SUBMITTED`, `ATTEMPTING`, `BACKOFF_WAIT`, `BUDGET_EXHAUSTED`, `DISPATCHED`, `TERMINAL_EXHAUSTED`).
- **Algorithm**: Enforce deterministic lifecycle states:
  - `SUBMITTED` -> `ATTEMPTING`: Attempt 1 launched with `Idempotency-Key`.
  - `ATTEMPTING` -> `DISPATCHED`: Success HTTP 201 received; tokens credited.
  - `ATTEMPTING` -> `BACKOFF_WAIT`: Transient error encountered, retry budget token available, attempt < 3.
  - `ATTEMPTING` -> `BUDGET_EXHAUSTED`: Transient error encountered, but token balance <= 0; fast-fail HTTP 503 emitted.
  - `ATTEMPTING` -> `TERMINAL_EXHAUSTED`: Attempt == 3 failed or permanent 4xx received; route to dead-letter audit store.
  - `BACKOFF_WAIT` -> `ATTEMPTING`: Sleep interval elapsed, deadline remaining > 500 ms; issue next attempt.
- **Outputs**: Published transition telemetry events to Kafka topic `auth.sms.lifecycle.v1`.
- **Owner**: Sarah Chen (Telephony Reliability Lead).
- **Failure Handling**: Any illegal state transition (e.g. attempting retry after `TERMINAL_EXHAUSTED`) trips runtime assertion and halts worker thread.
- **Verification**: State transition audit test `tests/lifecycle/test_retry_state_machine.py` (Freshness: 2026-09-15) validating zero state leakage across 5,000 concurrent flows.

#### 4. Recovery Test [MC-RT-01]
- **Inputs**: Synthetic telephony fault injection: 1,200 req/sec primary traffic, downstream HTTP 429 injected on 60% of requests for 45 seconds, returning to baseline (< 1% error, < 120 ms latency) at $T = 45\text{s}$.
- **Algorithm**:
  1. Fault injection: Inject 429 rate limits; verify client token bucket throttles retries within 10% overhead cap (outbound traffic <= 1,320 req/sec).
  2. Verify zero IP blacklist bans from telephony provider.
  3. Verify recovery timing: Upon fault removal at $T = 45\text{s}$, retry attempts drop to baseline (< 2 retries/sec) within 2.5 seconds.
  4. Assert 100% of exhausted dispatches cleanly arrive in fallback queue without lost records.
- **Outputs**: Automated recovery timing benchmark report confirming recovery duration <= 3.0 seconds.
- **Owner**: Marcus Vance (Security Infrastructure Architect).
- **Failure Handling**: If recovery timing exceeds 5.0 seconds post fault removal, trigger alert `ALERT_RETRY_RECOVERY_LAG` and reset token bucket.
- **Verification**: Chaos recovery test script `scripts/drills/run_telephony_retry_recovery_drill.sh` (Freshness: 2026-09-15) exiting 0 with logged timing metrics.

---

### Adversarial Cases and Routing

#### 1. Reject Infinite Retry [ADV-IR-01]
- **Vulnerability**: Client configuration or upstream callers configuring unconstrained, zero-delay, or infinite retry loops when receiving downstream HTTP 429/503 responses, generating thundering herd collapses.
- **Adversarial Mechanism**: In incident INC-2901, unconstrained immediate client retries multiplied outbound traffic by 8x (9,600 req/sec), triggering vendor security firewalls and blacklisting production IP ranges for 35 minutes.
- **Enforcement & Diagnostic**: Strict attempt ceilings and token bucket admission. The retry interceptor rejects any attempt beyond 3 total dispatches with RFC 9457 error `ERR_RETRY_LIMIT_EXCEEDED` and halts loop execution. Upstream clients attempting rapid retries receive HTTP 429 with header `Retry-After: 30`.
- **Forbidden Output Behavior**: The system is strictly forbidden from issuing uncounted, un-jittered, or infinite retries, and forbidden from executing retries when remaining deadline budget is <= 500 ms.

#### 2. Reject Shared Failure Domain [ADV-FD-01]
- **Vulnerability**: Routing retries through the identical degraded external telephony connection pool, shared DNS cache, or proxy gateway that is currently saturated, causing retries to exacerbate connection leaks.
- **Adversarial Mechanism**: When telephony provider edge encounters TLS handshake stalls, retrying on the same shared HTTP connection pool exhausts all 150 gateway connection leases, starving unrelated internal SMS notification services.
- **Enforcement & Diagnostic**: Connection isolation and independent egress routes. 2FA verification calls utilize an isolated Netty connection pool (`pool-sms-auth`, max 50 connections) completely decoupled from general marketing/notification pools. Cross-pool permit borrowing triggers diagnostic `ERR_SHARED_FAILURE_DOMAIN_LEAK`.
- **Forbidden Output Behavior**: The system is strictly forbidden from sharing connection pools, thread executors, or rate-limit quotas between critical 2FA authentication SMS dispatches and non-critical notification services.

#### 3. Reject Untested Compensation [ADV-UC-01]
- **Vulnerability**: Triggering automatic unverified fallback routes (e.g. silently sending authentication codes via unencrypted email or secondary unvetted VoIP providers) upon retry exhaustion without cryptographic integrity checks and security verification.
- **Adversarial Mechanism**: During SMS vendor brownouts, an unverified fallback service routed 2FA OTP codes via plain text webhook to a third-party analytics collector, creating severe credential exposure.
- **Enforcement & Diagnostic**: Fallback compensation must route strictly to verified, encrypted channels (such as audited push notification to registered authenticators or hardened fallback carrier routes). Attempting to route OTP tokens to unverified channels triggers diagnostic `ERR_UNTESTED_COMPENSATION_BLOCKED` and halts delivery.
- **Forbidden Output Behavior**: The system is strictly forbidden from routing authentication secrets to uncertified fallback vendors or unencrypted endpoints upon primary retry exhaustion.

---

### Invariants and Contracts

    Three-Attempt Hard Ceiling [INV-RTY-01]
      No request may be retried more than 2 times (3 total attempts). After the 3rd attempt,
      the request must terminate and emit a failure signal.

    Strict Prohibition of Immediate Retries [INV-RTY-02]
      Immediate un-jittered retries (sleep = 0) are strictly forbidden. All retry attempts must
      incorporate randomized full jitter to prevent thundering-herd synchronization.

    Non-Retryability of 4xx Client Errors [INV-RTY-03]
      HTTP 4xx client errors (excluding 429) must never be retried. Supplying malformed phone
      numbers or bad credentials must fail fast on attempt 1.

    Ten-Percent Traffic Budget Invariant [INV-RTY-04]
      Under no circumstance may retry volume exceed 10% of primary request volume over any 60-second
      rolling window. When the budget is depleted, retries must cease immediately.

    Stable Idempotency Scoping [INV-RTY-05]
      Every retry attempt must preserve the exact original Idempotency-Key header:
      SMS-2FA-{user_id}-{login_session_id}. Retrying with new keys is strictly forbidden.

## Explicit Unknowns

- Telephony carrier transit latency variance during regional cellular network outages (G-1).
- Telephony vendor server-side queue retention duration for accepted but delayed MT-SMS messages (G-2).

## Traceability

| Claim | Classification | Source | Freshness |
|---|---|---|---|
| Peak 1,200 auth requests/sec | provided | Traffic profile intake | Current |
| Incident INC-2901 8x retry storm (9,600 TPS) | provided | Post-mortem evidence | Historical |
| 35-minute vendor IP ban lockout | provided | Historical incident record | Historical |
| 3-attempt ceiling and full-jitter backoff | decided | Marcus Vance & Sarah Chen | 2026-09-15 |
| Base 100 ms, multiplier 2.0, cap 3,000 ms | decided | Architectural decision MC-BJ-01 | 2026-09-15 |
| 10% global retry budget allocation | decided | Infrastructure Platform Policy | 2026-09-15 |
| Isolated Netty connection pool (max 50 conns) | decided | Architectural invariant ADV-FD-01 | 2026-09-15 |
| Reject unverified fallback routing | decided | Architectural invariant ADV-UC-01 | 2026-09-15 |


## Verification

No validator was supplied, so no command was run.

Reviewer self-check against retry architecture contracts:
- **Failure Mode Matrix**: PASS. Covers HTTP 429, 503, 504, connection drops, and permanent 4xx errors.
- **Policy Enforcement**: PASS. AWS Full Jitter formula, 3-attempt ceiling, and 10% token bucket budget.
- **State Transition Integrity**: PASS. Deterministic state lifecycle with zero thread leakage.
- **Recovery Test Validation**: PASS. Fault injection drill demonstrates recovery in <= 2.5s with zero IP bans.
- **Adversarial Defenses**: PASS. Explicit rejection of infinite retry, shared failure domain, and untested compensation.
- **Idempotency Safety**: PASS. Strict invariant INV-RTY-05 preserves stable idempotency keys across retries.

## Open Decisions

- `DEC-RTY-01`: Sarah Chen to determine whether secondary fallback telephony providers (e.g. Infobip/Vonage) should be engaged automatically when the primary vendor budget is exhausted (Owner: Sarah Chen).

## Next steps

1. Sarah Chen configures Resilience4j / Polly retry policy with full jitter and 10% token bucket in `auth-sms-gateway`.
2. SRE team sets up Datadog monitor alerting on `retry_budget_exhausted_total > 0`.
3. Conduct staging resilience test injecting 50% vendor HTTP 429 drops to verify retry volume stays <= 10%.