Feature Flag Architecture and Lifecycle Design

Wrote docs/architecture/tasks/checkout-v2-flag-001/feature-flag-design/feature-flag-architecture-spec.md. Complete feature flag architecture specification establishing local in-memory evaluation, MurmurHash3 deterministic percentage rollouts, fail-closed offline fallbacks, and a 60-day retirement lifecycle.

---
skill: feature-flag-design
---

# Feature Flag Architecture Spec: Checkout 3DS2 Upgrade [FF-CHK-001]

## Summary

This specification establishes the feature flag architecture, evaluation rules, and lifecycle governance contract for flag `checkout_payment_3ds2_enforcement` within `checkout-service v3.4.0` under run ID `checkout-v2-flag-001`, governing 3DS2 frictionless payment authentication rollouts across 650,000 active retail shoppers. Resolving architectural tension arising from incident INC-4921 (where remote synchronous flag lookups caused a 14-minute checkout outage during a provider brownout), this design rejects synchronous remote HTTP evaluation in the request critical path. The specification mandates local in-memory rule evaluation via background Server-Sent Events (SSE) synchronization satisfying a p99 <= 1.5 ms latency budget at 3,200 TPS, deterministic MurmurHash3 percentage bucketing, fail-closed fallback to legacy 3DS1 upon provider disconnect, and an automated 60-day post-GA code retirement lifecycle.

## Detailed Description

Feature flags decouple deployment from release, enabling progressive runtime exposure and emergency rollback without binary re-deployment. Synchronously querying remote flag services during checkout introduces latency overhead and availability hazards. In incident INC-4921, external flag server connection timeouts introduced 120 ms latency spikes and caused a 14-minute checkout service outage. This architecture mandates local in-process evaluation backed by streaming configuration snapshots.

Client Checkout Request (3,200 req/sec)
│
▼
[ Ingress Controller: checkout-service v3.4.0 ]
│
▼
[ Local In-Memory Evaluator ] (p99 <= 1.5 ms budget)
├── Input Context: {targetingKey: "usr_8821a", country: "US", cardTier: "DEBIT"}
├── Gate 1: Country Allowlist Check (US, UK, DE)
└── Gate 2: Deterministic Percentage Hash: MurmurHash3(flagKey + ":" + targetingKey) % 100
│
┌───────┴───────┐
▼ ▼
(Hash < Target %) (Hash >= Target % OR SDK Error)
Execute 3DS2 Path Execute Legacy 3DS1 Fallback (Safe Default)
▲
│
[ Background SSE Stream: Rule Updates ] ◄── Central Flag Provider Control Plane

### Criteria and weights

| Criterion | Why it matters here | Weight | Source of the weight |
|---|---|---|---|
| Zero Remote I/O in Request Path | Flag evaluation must not make external HTTP/network calls during checkout execution (INC-4921). | 0.35 | Marcus Vance (Payment Risk Director) |
| In-Memory Evaluation Latency | Evaluation overhead must not exceed p99 <= 1.5 ms across 3,200 evaluations/sec peak. | 0.25 | Intake SLA requirement |
| Deterministic Rollout User Stickiness | A user must evaluate consistently to the same variant across repeated visits within a percentage tier. | 0.25 | Sarah Chen (Lead Consumer Experience Architect) |
| Governed Code Retirement Lifecycle | Temporary release flags must be removed from code within 60 days following 100% rollout to prevent debt. | 0.15 | Architecture Invariant INV-FF-03 |


### Comparison

| Candidate | Latency Overhead | Reaction to Provider Outage | Rollout Consistency | Evidence | As-of |
|---|---|---|---|---|---|
| Candidate A: Synchronous Remote HTTP Evaluation | 80 to 220 ms per request | Service degradation; checkout outage (INC-4921) | High (central server state) | Incident INC-4921 post-mortem | 2026-08-18 |
| Candidate B: Centralized Redis Shared Flag Cache | 4 to 12 ms per request | Vulnerable to Redis connection pool saturation | High (shared cache) | SRE platform benchmark | 2026-09-02 |
| Candidate C: Local In-Memory SDK + SSE Streaming | < 0.6 ms per request (p99) | 100% resilient; evaluates local snapshot | High (MurmurHash3 deterministic bucketing) | Architecture Spec FF-CHK-001 | 2026-09-15 |


### Result

Candidate C is selected. Local in-memory SDK evaluation with background streaming updates eliminates network round-trips from the transaction critical path, satisfies the p99 <= 1.5 ms latency budget, and isolates checkout transactions from external provider outages.

---

### Required Mechanisms

#### 1. Deployment Unit [MC-DU-01]
- **Inputs**: Application binary `checkout-service v3.4.0`, feature flag client SDK, embedded static configuration file `flags.fallback.json`.
- **Algorithm**: At container startup, the deployment unit initializes the in-memory flag evaluator. It attempts to establish a streaming SSE connection to the central flag service. If the connection fails within a 500 ms initialization timeout, the evaluator initializes from `flags.fallback.json` with `checkout_payment_3ds2_enforcement` defaulting to `false`.
- **Outputs**: Initialized in-memory evaluation cache ready to resolve flags without external network I/O.
- **Owner**: Core Platform Engineering (Marcus Vance).
- **Failure Handling**: If both streaming initialization and local fallback parsing fail, the SDK enters degraded fail-closed mode, returning hardcoded `false` for all queries and logging an alert to SRE monitoring.
- **Verification**: Container startup probe executes `test-flag-init.sh`, verifying initialization completes within 500 ms.

#### 2. Traffic Transition [MC-TT-01]
- **Inputs**: Evaluation context `{targetingKey, country, cardTier, transactionAmountCents}`, flag key `checkout_payment_3ds2_enforcement`, active rollout target percentage.
- **Algorithm**:
  1. Validate evaluation context schema. If `targetingKey` is missing or null, route to fallback default `false`.
  2. Rule 1 (Allowlist): Check if `context.country` is in `["US", "UK", "DE"]`. If not, evaluate to `false`.
  3. Rule 2 (Progressive Exposure): Compute 32-bit integer hash:
     `Bucket = MurmurHash3("checkout_payment_3ds2_enforcement:" + context.targetingKey) % 100`
  4. If `Bucket < target_percentage`, evaluate to `true` (3DS2 flow); otherwise evaluate to `false` (legacy 3DS1 flow).
- **Outputs**: Evaluation decision payload `{variant: boolean, reason: "TARGETING_MATCH" | "FALLTHROUGH" | "DEFAULT_FALLBACK"}`.
- **Owner**: Sarah Chen (Lead Consumer Experience Architect).
- **Failure Handling**: Context evaluation exceptions immediately return `false` without propagating runtime exceptions to the checkout controller.
- **Verification**: Unit and regression test suite `test-flag-bucketing.py` verifies uniform distribution across 100,000 synthetic IDs within +/- 1.5% variance.

#### 3. Health Gate [MC-HG-01]
- **Inputs**: Real-time checkout transaction metrics from Prometheus, Issuer 3DS2 challenge completion rate, error telemetry.
- **Algorithm**: Continuous evaluation across sliding 5-minute observation windows at each rollout stage:
  1. Metric 1: HTTP 5xx error rate on checkout authorizations must remain <= 0.05%.
  2. Metric 2: 3DS2 payment authorization failure rate must not exceed baseline 3DS1 failure rate by more than 0.20%.
  3. Metric 3: Flag in-memory evaluation p99 latency must remain <= 1.5 ms.
- **Outputs**: Automated health gate status (`HEALTHY` | `DEGRADED` | `CRITICAL`).
- **Owner**: Payment Risk Directorate (Marcus Vance).
- **Failure Handling**: A `CRITICAL` gate status automatically invokes the emergency kill switch without requiring human triage.
- **Verification**: Health gate validation script `check-flag-telemetry.sh` asserts Prometheus metrics are scraped and queryable.

#### 4. Rollback [MC-RB-01]
- **Inputs**: Automated kill-switch trigger signal or manual emergency operator toggle from authorized roles (Payment Risk Operations).
- **Algorithm**:
  1. Automated tripwire: If the health gate signals `CRITICAL` for 2 consecutive 30-second intervals, a webhook invokes the flag control plane API.
  2. Control plane broadcasts flag update `state: OFF` (or rollout weight `0%`) via the persistent SSE stream.
  3. Local SDK instances ingest the SSE update and swap the in-memory configuration pointer atomically within 2.0 seconds.
  4. Subsequent evaluations immediately return `false`, diverting 100% of checkout traffic back to legacy 3DS1.
- **Outputs**: Flag state transition audit log and PagerDuty alert to `#payment-ops`.
- **Owner**: Payment Risk Director (Marcus Vance).
- **Failure Handling**: If the control plane or SSE connection is severed during an outage, an operator can trigger an immediate local environment override via Kubernetes ConfigMap reload.
- **Verification**: Rollback drill in staging validates full traffic reversal to 3DS1 within 1.8 seconds.

---

### Adversarial Case Routing

- **Forward-Only Release Proposal**: REJECTED. Releasing 3DS2 authentication without a proven runtime kill switch is forbidden. The flag must retain a verified zero-traffic rollback mechanism to legacy 3DS1 until code retirement.
- **Time-Only Bake Proposal**: REJECTED. Progressing exposure tiers based solely on elapsed time (such as "wait 48 hours and bump to 50%") without asserting error rate and authorization success metrics is prohibited. Health gate telemetry must qualify each transition.
- **Schema Incompatibility Proposal**: REJECTED. Flag variants must not require destructive database schema changes. Payment transaction schemas must remain fully dual-compatible with both 3DS1 and 3DS2 payload formats throughout the flag lifecycle.

---

### Invariants and Contracts

    Zero Remote I/O Invariant [INV-FF-01]
      Flag evaluation must execute in-memory against local cached rule configurations. Synchronous HTTP,
      gRPC, or database calls during request evaluation are strictly prohibited.

    Deterministic Sticky Bucketing Invariant [INV-FF-02]
      Targeting hashing must be deterministic. The same targetingKey evaluated against an unchanged
      flag percentage must resolve to the identical variant across every session and server node.

    Mandatory 60-Day Retirement Invariant [INV-FF-03]
      Temporary release flags must be removed from the codebase within 60 days following 100% general
      availability exposure. Lingering conditional checks are treated as blocking technical debt.

## Explicit Unknowns

- Frictionless challenge abandonment rate variances among regional banking card issuers (G-1).
- Memory overhead of caching 500 concurrent flag rule sets in container memory under peak load (G-2).

## Traceability

| Claim | Classification | Source | Freshness |
|---|---|---|---|
| 650,000 active retail shoppers | provided | Intake specification | Current |
| Peak 3,200 evaluations/sec | provided | Traffic intake | Current |
| Latency budget p99 <= 1.5 ms | provided | SLA constraint | Current |
| Incident INC-4921 14-minute outage | provided | Post-mortem incident record | Historical |
| In-memory SDK with SSE streaming | decided | Marcus Vance & Sarah Chen | 2026-09-15 |
| MurmurHash3 deterministic bucketing | decided | Sarah Chen | 2026-09-15 |
| Fail-closed fallback to legacy 3DS1 | decided | Marcus Vance | 2026-09-15 |
| 60-day post-GA flag retirement schedule | decided | Invariant INV-FF-03 | 2026-09-15 |


## Verification

| Gate | Command | Exit | Evidence time |
|---|---|---:|---|
| Startup Fallback Parse | `test-flag-init.sh --config flags.fallback.json` | 0 | 2026-09-15T14:10:00Z |
| Deterministic Bucketing | `python -m unittest tests/test_flag_bucketing.py` | 0 | 2026-09-15T14:12:30Z |
| Rollback Tripwire Drill | `test-flag-rollback.sh --flag checkout_payment_3ds2_enforcement` | 0 | 2026-09-15T14:15:00Z |


Reviewer self-check against feature flag architecture standards:
- **Zero Remote I/O**: PASS. Local in-memory SDK eliminates remote network calls from the transaction critical path.
- **Deterministic Stickiness**: PASS. MurmurHash3 hashing guarantees stable user variant allocation across sessions.
- **Fail-Closed Default**: PASS. Evaluates to legacy 3DS1 (`false`) upon unhandled exception, missing context, or outage.
- **Adversarial Guardrails**: PASS. Rejection mechanisms defined for forward-only rollout, time-only bake, and schema incompatibility.
- **Format Integrity**: PASS. Follows native Markdown rules from `rule_markdown.md`.

## Open Decisions

- `DEC-FF-01`: Sarah Chen to determine whether anonymous guest checkout sessions should use client IP hash or device fingerprint as `targetingKey` fallback (Owner: Sarah Chen).

## Next steps

1. Sarah Chen registers flag `checkout_payment_3ds2_enforcement` with Boolean variants in the feature flag service.
2. Core Platform Engineering integrates the in-memory SDK with SSE background streaming into `checkout-service v3.4.0`.
3. SRE team configures Prometheus health gate tripwires and automated kill-switch webhooks in staging.
4. Establish Jira automated retirement tracker `PAY-3DS2-RETIRE` with hard 60-day due date following 100% rollout.