Dependency Circuit Breaker Design

Wrote docs/architecture/tasks/credit-check-cb-001/circuit-breaker-design/circuit-breaker-contract.md. Complete circuit breaker contract specifying count-based sliding windows, 50% failure threshold, 2,000 ms slow-call tripwires, 30-second open durations, and fallback underwriting queues.

---
skill: circuit-breaker-design
---

# Circuit Breaker Contract: Credit Bureau External Integration [CB-CREDIT-001]

## Summary

This specification establishes the circuit breaker resilience contract for `loan-origination-service v2.4` under run ID `credit-check-cb-001`, governing remote calls to the external Credit Bureau API across 220 peak requests/second. Following incident INC-3120 (where remote 15-second timeouts and aggressive retries exhausted all 200 application worker threads for 28 minutes), this design rejects simple un-gated timeouts. The contract specifies Resilience4j circuit breaking with a 100-call count-based sliding window, a 50% failure rate threshold, a slow-call duration threshold (>= 2,000 ms at 60% rate), a 30-second OPEN state wait window, and 10 HALF_OPEN trial executions before closure. When OPEN, requests fail fast in < 2 ms, immediately routing loan applications to an asynchronous underwriter review queue.

## Detailed Description

Synchronous dependencies on third-party SaaS APIs present severe availability risks. When an external partner experiences latency degradation, client threads block indefinitely waiting on TCP read timeouts, cascading backwards into caller thread starvation and gateway collapse.

Incoming Loan Application (220 req/sec)
│
▼
[ Resilience4j Circuit Breaker: credit_bureau_cb ]
├── State: CLOSED (Normal: failure < 50%, slow < 60%) ──► Execute HTTP Call
├── State: OPEN (Tripped: fail-fast < 2 ms) ────────────► Route to Fallback Queue
└── State: HALF_OPEN (10 Trial Probes) ─────────────────► Evaluate Health
│
▼ (Call Execution)
[ External Credit Bureau Gateway ]
├── Success (< 2s) ──► Record Success in Window
├── Timeout (> 2s) ──► Record Slow Call
└── HTTP 5xx ────────► Record Failure

### Criteria and weights

| Criterion | Why it matters here | Weight | Source of the weight |
|---|---|---|---|
| Caller Thread Pool Protection | Blocking worker threads behind third-party timeouts collapses loan origination (INC-3120). | 0.40 | Marcus Vance (Backend Architect) |
| Rapid Fail-Fast Reaction (< 2 ms) | Tripped circuits must shed load immediately without holding client connections. | 0.25 | Intake SLA requirement |
| Downstream Partner Recovery Courtesy | Continuing to pump 220 req/sec into a degraded credit bureau prevents upstream recovery. | 0.20 | Sarah Chen (Credit Risk) |
| Deterministic Fallback Underwriting | Applications must not be discarded; loan files must queue safely for offline review. | 0.15 | Credit Risk Operations |


### Comparison

| Candidate Strategy | Failure Detection Mechanism | State Recovery Trial | Blast Radius to Worker Pool | Residual Outage Risk |
|---|---|---|---|---|
| Option A: Simple Timeout (3s) + 3 Retries | Local HTTP client socket timeout | None (every request calls remote) | Catastrophic: 220 req/s * 3 retries locks 660 threads | Critical: Recreates INC-3120 thread lockup. |
| Option B: Time-Based Moving Average (60s) | Sliding 60-second time bucket | Abrupt full-traffic reopen | High: Sudden flood of 220 TPS re-trips recovery | High: Oscillates between open and closed state. |
| Option C: Count-Based Sliding Window + Half-Open (Chosen) | 100-call circular buffer (50% fail, 60% slow) | 10 bounded trial calls in HALF_OPEN | Minimal: Sheds 100% load during 30s OPEN state | Minimal: Smooth recovery probe, zero thread lock. |


### Result

Option C is selected. A count-based sliding window of 100 calls captures latency spikes and errors, isolating caller threads via immediate fail-fast exceptions.

---

### Required Mechanisms

#### 1. Task Contract & Dependency Boundary [MC-TC-01]
- **Protected Operation**: `GET /v2/credit-reports/{tax_id}`
- **Execution Boundary**: Ingress wrapper `CreditBureauClientWrapper`.
- **Configured Timeout**: Socket read timeout strictly capped at 2,500 ms.
- **Fail-Fast Error**: `CallNotPermittedException` emitted when circuit is OPEN, completing in < 2 ms.

#### 2. Sliding Window & Threshold Mathematics [MC-SW-01]
- **Window Type**: Count-based sliding window (N = 100 recorded calls).
- **Minimum Number of Calls**: 20 calls must be recorded before thresholds are evaluated.
- **Failure Rate Threshold**: >= 50.0% (HTTP 5xx, `SocketTimeoutException`, `ConnectException`).
  $$\text{FailureRate} = \frac{\sum \text{Failed Calls}}{\text{Total Recorded Calls}} \times 100 \ge 50\%$$
- **Slow Call Duration Threshold**: Execution duration >= 2,000 ms.
- **Slow Call Rate Threshold**: >= 60.0% of calls in window taking >= 2,000 ms.

#### 3. State Transition Matrix [MC-ST-01]

| From State | Trigger Condition | To State | Duration / Concurrency Constraint |
|---|---|---|---|
| `CLOSED` | Failure rate >= 50% OR Slow call rate >= 60% | `OPEN` | Immutably locked for 30 seconds. |
| `OPEN` | Elapsed time > 30.0 seconds | `HALF_OPEN` | Transitions automatically upon next incoming request. |
| `HALF_OPEN` | 10 trial calls execute: failure rate < 50% | `CLOSED` | Normal traffic resumes; window resets. |
| `HALF_OPEN` | 10 trial calls execute: failure rate >= 50% | `OPEN` | Re-trips to OPEN for another 30 seconds. |


#### 4. Fallback & Degradation Specification [MC-FB-01]
When `CallNotPermittedException` or remote timeout occurs:
1. Interceptor suppresses exception from caller view.
2. Checks local Redis cache for recent credit score (< 30 days old).
   - If cache hit: return cached tier with header `X-Credit-Source: CACHED-FALLBACK`.
   - If cache miss: enqueue loan application to PostgreSQL table `manual_underwriting_queue` with status `QUEUED_FOR_OFFLINE_REVIEW`.
3. Return HTTP 202 Accepted to customer: *"Credit verification is pending manual review. Your application status will update within 4 hours."*

---

### Invariants and Contracts

    Zero Worker Thread Starvation [INV-CB-01]
      When the circuit breaker is OPEN, no outbound network sockets may be opened. Requests must
      fail fast and enter the fallback path within 5 ms, consuming zero execution thread leases.

    Bounded Half-Open Probing [INV-CB-02]
      In the HALF_OPEN state, concurrent outbound calls are hard-capped at 10 requests. All other
      concurrent callers during the trial phase must continue to receive immediate fallback routing.

    Deterministic Error Accounting [INV-CB-03]
      Client validation errors (HTTP 400 Bad Request, 404 Not Found) must be classified as benign
      and excluded from the failure rate calculation. Only 5xx and timeouts count as failure events.

## Explicit Unknowns

- Credit bureau network socket behavior during upstream Cloudflare DDoS protection challenges (G-1).
- Rate of manual underwriter ticket clearance during sustained multi-hour bureau outages (G-2).

## Traceability

| Claim | Classification | Source | Freshness |
|---|---|---|---|
| Peak 220 requests/sec | provided | Traffic intake | Current |
| Bureau latency normal 350 ms, brownout 15,000 ms | provided | Incident metric logs | Historical |
| Incident INC-3120 28-minute thread lockup | provided | Post-mortem evidence | Historical |
| Count-based window 100 calls, 50% fail rate | decided | Marcus Vance & Sarah Chen | 2026-09-15 |
| Slow call duration >= 2,000 ms at 60% rate | decided | Architectural decision MC-SW-01 | 2026-09-15 |
| 30-second OPEN state duration | decided | Resilience Policy | 2026-09-15 |


## Verification

No validator was supplied, so no command was run.

Reviewer self-check against circuit breaker architecture standards:
- **Threshold Completeness**: PASS. Explicit 50% failure rate and 60% slow-call duration thresholds.
- **State Recovery Bounds**: PASS. 30s open wait duration with 10-call half-open trial quota.
- **Fail-Fast Safety**: PASS. Bypasses remote network calls during OPEN state in < 2 ms.
- **Fallback Integrity**: PASS. Dual fallback (Redis cache hit -> offline underwriter queue).

## Open Decisions

- `DEC-CB-01`: Sarah Chen to determine whether manual underwriting queue tickets should trigger automated SMS notifications to loan applicants (Owner: Sarah Chen).

## Next steps

1. Marcus Vance configures Resilience4j `CircuitBreakerRegistry` bean in `loan-origination-service`.
2. Platform team sets up Datadog monitor alerting on `resilience4j_circuitbreaker_state{state="open"}`.
3. Conduct staging game day simulating 100% bureau timeout to verify 100% loan capture in fallback queue.