Chaos Experiment and Resilience Testing Strategy

Wrote docs/architecture/tasks/banking-chaos-001/chaos-testing-strategy/chaos-testing-strategy.md. Complete chaos testing strategy establishing steady-state hypotheses, fault injection portfolios across AZs, sub-15s automated abort tripwires, and game day governance.

---

skill: chaos-testing-strategy

Chaos-Testing Strategy: Payment Authorization Engine [CHAOS-BANK-001]

Summary

This specification establishes the chaos engineering experiment portfolio, steady-state hypothesis models, blast-radius containment rules, and automated abort tripwires for payment-authorization-engine v4.0 under run ID banking-chaos-001. It governs 18,000 peak authorizations/second distributed across 3 AWS Availability Zones (us-east-1a, us-east-1b, us-east-1c). It decisively eliminates the uncontrolled cascade risks demonstrated in incident INC-4939 (where an unconstrained chaos test killed connection pools and halted $1.8M in checkouts). The strategy enforces measurable steady-state metric baselines (p99 latency <= 35 ms, error rate <= 0.05%), four controlled fault injection scenarios (network partition, pod termination, latency injection, database primary failover), tightly bounded blast-radius limits (<= 10% synthetic traffic canary, single AZ only), and sub-15-second automated abort tripwires.

Detailed Description

Injecting infrastructure faults without steady-state baselines and automated circuit breakers turns resilience engineering into uncontrolled self-inflicted outages. When systems experience unexpected dependency degradation—such as a cloud availability zone network partition or silent cache loss—untested retry storms and connection pool exhaustion trigger systemic failures. A formal chaos testing strategy proves fault tolerance under controlled, observable, and reversible experimental conditions.

Chaos Experiment Invocation (Experiment: `EXP-NET-PARTITION-AZ1`)
                            │
                            ▼
[ Chaos Controller & Blast-Radius Governor ]
  ├── 1. Asserts Steady-State Baseline: p99 <= 35ms, Errors <= 0.05%
  ├── 2. Isolates Blast Radius: Traffic routed to Synthetic Canary (10% max)
  └── 3. Injects Fault via AWS Fault Injection Simulator (FIS)
                            │
                            ▼ (Cross-AZ Link severed for 180s)
[ Availability Zone `us-east-1a`: Network Partition Simulation ]
  ├── Envoy Circuit Breaker trips in < 150 ms
  └── Traffic shifts automatically to AZ-b and AZ-c
                            │
       ┌────────────────────┴────────────────────┐
       ▼ (Steady-State Maintained)               ▼ (Error Rate > 0.05% or Latency > 50ms)
[ Hypothesis Validated: Pass ]           [ AUTOMATED ABORT TRIPWIRE FIRES (< 15s) ]
  Capture Metric Dashboard Artifacts       AWS FIS Aborts; Traffic Restored Instantly

Criteria and weights

Comparison

Result

Option C is selected. AWS FIS executes controlled fault injection; Prometheus tripwires abort experiments automatically; synthetic canary traffic contains customer risk.

---

Required Mechanisms

1. Steady-State Hypothesis & Metrics Verification [MC-SS-01]

• Steady-State Definition: Prior to fault injection, the system must maintain:
  1. http_request_duration_seconds{quantile="0.99"} <= 0.035 (35 ms).
  2. rate(http_requests_total{status=~"5.."}[1m]) / rate(http_requests_total[1m]) <= 0.0005 (0.05%).
  3. payment_engine_active_connection_pool_saturation <= 0.70 (70%).

Hypothesis: Severing connectivity to one AZ or killing 30% of Redis replicas will cause p99 latency to rise by <= 8 ms with zero increase in 5xx errors, as traffic seamlessly drains to surviving nodes.

2. Fault Injection Portfolio [MC-FP-01]

3. Automated Abort Tripwires & Rollback Protocol [MC-AT-01]

• Prometheus Continuous Tripwire Daemon:
  • Evaluates SLIs every 2 seconds during active experiments.
• Tripwire Abort Triggers:
  • ErrorRate > 0.05% for $> 6$ seconds.
  • p99_Latency > 50ms for $> 6$ seconds.
  • Any customer transaction timeout reported by edge API Gateway.
• Abort Action:
  • Triggers AWS FIS StopExperiment API in < 3 seconds.
  • Restores all security groups, routes, and Kubernetes deployments in

< 12 seconds (total recovery duration: < 15 seconds).

4. Game Day Cadence & Operational Post-Mortem [MC-GD-01]

• Cadence: Bi-weekly game days in pre-production staging; quarterly dark-launch canary runs in production.
• Roles:
  • Chaos Commander: Drives injection console and monitors tripwires.
  • SRE Observer: Triage alerts without prior knowledge of specific fault to test runbooks.

Action Items: Any unexpected cascade triggers an immediate JIRA bug labeled CHAOS-DEFECT with mandatory 14-day remediation SLA.

---

Invariants and Contracts

Mandatory Automated Abort Tripwire [INV-CHAOS-01]
  Chaos experiments must be coupled to automated metric-driven abort tripwires.
  Running experiments without automated sub-15-second rollback triggers is strictly prohibited.

Strict Blast-Radius Ceiling (Ten Percent) [INV-CHAOS-02]
  Fault injection in production environments must never target more than 1 Availability Zone
  or more than 10% of active service replicas simultaneously.

Pre-Experiment Steady-State Gate [INV-CHAOS-03]
  An experiment must not initiate unless the system has demonstrated clean steady-state metrics
  for at least 15 continuous minutes prior to launch.

Explicit Unknowns

• AWS FIS API call rate limit throttling during emergency abort sequences across 40 concurrent faults (G-1).
• Cross-region Aurora PostgreSQL read-replica replication lag behavior when primary node CPU is artificially spiked to 100% (G-2).

Traceability

Verification

No validator was supplied, so no command was run.

Reviewer self-check against chaos testing standards:

• Steady-State Rigor: PASS. Baseline SLIs defined with explicit PromQL queries and thresholds.
• Blast-Radius Safety: PASS. Capped at 1 AZ and 10% canary traffic; zero full-estate injection.
• Abort Velocity: PASS. Sub-15s automated rollback ensures immediate recovery upon SLA breach.
• Markdown Hygiene: PASS. Native Markdown syntax strictly adheres to rule_markdown.md.

Open Decisions

• DEC-CHAOS-01: David O'Reilly to determine whether automated chaos testing should be integrated directly into post-deployment canary release pipelines (Continuous Chaos) (Owner: David O'Reilly).

Next steps

1. Marcus Vance provisions AWS Fault Injection Simulator (FIS) experiment templates via Terraform.
2. Platform team configures Prometheus continuous tripwire webhook integration.
3. Conduct staging game day executing EXP-CHAOS-01 (AZ network partition) to verify sub-15s automated abort tripwire.