Workload and Load-Testing Strategy

Wrote docs/architecture/tasks/banking-load-001/load-testing-strategy/load-test-scenario-spec.md. Complete load test scenario specification establishing open-loop arrival models, distributed k6 execution clusters, latency SLA verification, and database saturation gates.

---
skill: load-testing-strategy
---

# Load Test Scenario Spec: Black Friday Core Checkout [LOAD-BANK-001]

## Summary

This specification establishes the executable workload modeling architecture, load test scenario specification, and capacity qualification framework for `core-checkout-service v3.4` under run ID `banking-load-001`. It prepares the banking platform for annual peak retail events, sustaining an open-loop peak arrival rate of 28,000 requests/second and a sustained plateau of 20,000 requests/second across 4 hours. It decisively eliminates the false confidence demonstrated in incident INC-3890 (where unrealistic closed-loop virtual user testing failed to anticipate coordinated checkout spikes, causing total database connection pool exhaustion and dropping $2.4M in retail transactions). The strategy enforces an open-loop Poisson arrival rate model, distributed load execution via k6 on AWS EKS, a realistic transactional mix (70% idempotent read / 30% mutating write), strict p95 <= 45 ms and p99 <= 85 ms latency SLAs, and an automated capacity qualification gate blocking releases if Aurora PostgreSQL database connection pool saturation exceeds 75%.

## Detailed Description

Relying on traditional "Virtual User" (VU) closed-loop load generators introduces coordinate omission bias. In a closed loop, virtual users wait for a server response before issuing the next request; when the server slows down under stress, the load generator artificially throttles its request dispatch rate, hiding severe queuing delays. An open-loop load generator schedules requests according to an arrival rate independent of server responsiveness, accurately reflecting production traffic during promotional surges.

Distributed Load Generator: k6 on AWS EKS (12 Worker Pods)
│
▼ (Open-Loop Arrival Rate: 28,000 req/sec)
[ Ingress Gateway: AWS NLB + Envoy Proxy ]
├── 1. Enforces TLS 1.3 Handshakes across 3 Availability Zones
└── 2. Distributes Ingress Traffic across 65 Checkout Service Pods
│
┌────────────────┴────────────────┐
▼ (70% Reads: Cart / Balance) ▼ (30% Mutating: Checkout / Authorize)
[ Redis Idempotency Cache ] [ Aurora PostgreSQL 16 (Primary) ]
├── Sub-2ms Read Response ├── Enforces Row Locks on Account Balances
└── Connection Pool Saturation < 50%└── Connection Pool Saturation Capped at 75%
│
▼
[ Prometheus Performance Oracle: Latency & Saturation Gates ]
Asserts: p95 <= 45ms, p99 <= 85ms, DB Pool <= 75%, 5xx Errors <= 0.01%

### Criteria and weights

| Criterion | Why it matters here | Weight | Source of the weight |
|---|---|---|---|
| Open-Loop Arrival Realism (Zero VU Bias) | Load dispatch must continue unabated during latency spikes to expose true queuing collapse (INC-3890). | 0.40 | David O'Reilly (Lead SRE Architect) |
| Transactional Payload Mix Fidelity | Mutating database writes (30%) generate row lock contention absent in pure read tests. | 0.30 | Elena Rostova (Head of Payment Ops) |
| Latency SLA Compliance (p95 <= 45ms, p99 <= 85ms) | Checkout latency degradation directly increases consumer cart abandonment rates. | 0.15 | Core Commercial Banking SLA |
| Database Resource Envelope Safety (< 75% Pool) | Exhausting connection pools crashes adjacent clearing microservices sharing the database cluster. | 0.15 | Enterprise Database Infrastructure Policy |


### Comparison

Record measured values with their date and version. A vendor claim is a claim, not a measurement — classify it as `provided`, not `observed`.

| Candidate | Arrival Model | Traffic Mix | Latency Under Load (p99) | Evidence | As-of |
|---|---|---|---|---|---|
| Option A: Single-Host Static VUs (Legacy) | Closed-Loop (500 VUs) | 100% Reads (`GET /balance`) | 18 ms (Hollow / Artificial) | Incident INC-3890 | 2026-09-15 |
| Option B: Distributed HTTP Replay | Log Replay | Historical Playback | 142 ms (Exceeds SLA) | Staging Trial TR-8812 | 2026-09-15 |
| Option C: Distributed Open-Loop k6 (Chosen) | Open-Loop Arrival Rate | 70% Read / 30% Write | 74 ms (Meets <= 85ms SLA) | Load Test Run LT-9901 | 2026-09-15 |


### Result

Option C is selected. Distributed k6 agents generate open-loop Poisson arrival traffic; synthetic merchant test tokens isolate financial side effects; automated gates verify database saturation.

---

### Required Mechanisms

#### 1. Risk [MC-RK-01]
- **Inputs**: Black Friday volume forecast (28,000 arrival req/sec), service dependency graph (`core-checkout-service` -> Aurora PostgreSQL 16, Redis 7.2, Kafka settlement topics), historical failure log from incident INC-3890.
- **Algorithm**: Workload risk evaluation engine calculating queueing amplification factor, database write lock contention probability, and connection pool exhaustion thresholds under un-paced arrivals.
- **Outputs**: Bounded risk profile establishing that un-throttled open-loop checkout spikes trigger catastrophic thread starvation if Aurora active connection saturation breaches 75%.
- **Owner**: David O'Reilly (Lead SRE Architect).
- **Failure Handling**: If traffic forecasting indicates arrival rates exceeding generator capacity or downstream connection ceilings, the scenario planner rejects the execution plan with diagnostic `ERR_LOAD_CAPACITY_RISK_BREACH`.
- **Verification**: Pre-flight capacity model check `python scripts/verify_load_capacity_envelope.py` exiting 0.

#### 2. Test Layer [MC-TL-01]
- **Inputs**: System topology, deployment specifications, network boundary definitions.
- **Algorithm**: Environment fidelity routing mapping test execution across four defined layers:
  - *Layer 1 (Pre-merge Microbenchmark)*: Sociable in-memory concurrency checks on local developers machines.
  - *Layer 2 (Component Seam Load)*: Single-service containerized endpoint saturation testing via isolated test harness.
  - *Layer 3 (Integrated Performance Qualification)*: Distributed open-loop load testing on multi-AZ staging cluster identical to production topology.
  - *Layer 4 (Production Canary Load)*: Off-peak synthetic probe verification on live ingress endpoints.
- **Outputs**: Scenario specification formally allocated to Layer 3 (Integrated Performance Qualification) with 12 dedicated k6 worker nodes.
- **Owner**: Elena Rostova (Head of Payment Operations).
- **Failure Handling**: Attempting to execute multi-service capacity qualification against Layer 1 or shared developer environments triggers diagnostic `ERR_INVALID_TEST_LAYER_TOPOLOGY`.
- **Verification**: Architecture runner rule `check_test_layer_alignment.sh` asserting execution environment matches Layer 3 prerequisites.

#### 3. Oracle [MC-OR-01]
- **Inputs**: Prometheus edge latency metrics (`http_req_duration`), Aurora connection pool gauges (`pg_stat_activity`), application error counters (`http_requests_total{status=~"5.."}`).
- **Algorithm**: Automated statistical multi-metric performance evaluator:
  - *Throughput & Goodput Oracle*: Asserts completed successful requests >= 27,990 req/sec during peak phase.
  - *Latency Distribution Oracle*: Evaluates sliding 1-minute histogram windows asserting `p95 <= 45.0ms` and `p99 <= 85.0ms`.
  - *Saturation Oracle*: Polls Aurora PostgreSQL active connection count every 5 seconds; asserts `active_connections / max_connections <= 0.75` (capped at 3,750 / 5,000).
  - *Error Budget Oracle*: Asserts HTTP 5xx error rate remains `<= 0.01%` across all test stages.
- **Outputs**: Deterministic run verdict: `QUALIFIED`, `SATURATED`, or `VIOLATED_SLA`.
- **Owner**: David O'Reilly (Lead SRE Architect).
- **Failure Handling**: Breaching any latency SLA or connection saturation ceiling for > 15 consecutive seconds triggers automated scenario abort via k6 REST API, logging diagnostic `ERR_LOAD_ORACLE_THRESHOLD_TRIPPED`.
- **Verification**: Continuous metric verification daemon `python scripts/check_prometheus_load_oracle.py` confirming telemetry query validity.

#### 4. Coverage Gate [MC-CG-01]
- **Inputs**: Pull request git diff, service deployment artifacts, peak readiness checklist.
- **Algorithm**: Pre-holiday deployment gate enforcing:
  1. Successful completion of 4-hour 20,000 req/sec sustained soak test with zero unhandled memory leaks or pool degradation.
  2. Successful completion of 30-minute 28,000 req/sec flash surge scenario meeting all latency distribution oracles.
  3. 100% scenario coverage across the four defined transactional endpoints.
- **Outputs**: Release readiness token `PERF-GATE-PASS-BF2026` permitting production rollout.
- **Owner**: Quality Engineering Guild and Payment SRE Team.
- **Failure Handling**: If the candidate build fails to maintain p99 <= 85 ms or breaches the 75% connection saturation limit, the CI gate terminates with exit code 1 and diagnostic `ERR_LOAD_COVERAGE_GATE_FAILED`.
- **Verification**: Gate check script `python scripts/verify_perf_gate.py --run-id banking-load-001` exiting 0.

---

### Adversarial Cases and Routing

#### 1. Reject Pyramid by Habit [ADV-PH-01]
- **Vulnerability**: Defaulting to the testing pyramid by habit (relying exclusively on thousands of microbenchmarks and isolated unit tests while omitting realistic high-concurrency multi-user load testing), leading teams to assume green unit builds guarantee capacity readiness.
- **Adversarial Mechanism**: In incident INC-3890, all checkout unit and integration tests passed with 100% assertions; however, when 20,000 concurrent arrival requests hit production, database connection pool exhaustion and thread starvation caused total system failure.
- **Enforcement & Diagnostic**: Production capacity qualification mandates executable open-loop load testing scenarios under full traffic concurrency. Submitting release qualification sign-offs supported only by unit or microbenchmark suites triggers diagnostic `ERR_PYRAMID_BY_HABIT_REJECTED` and halts release approval.
- **Forbidden Output Behavior**: The testing strategy is strictly forbidden from certifying production capacity based on unit test results, isolated single-user benchmarks, or component coverage metrics.

#### 2. Reject Assertion-Free Test [ADV-AF-01]
- **Vulnerability**: Executing volumetric load tests that pump millions of HTTP requests through a gateway but contain zero assertions on HTTP response bodies, database commit persistence, or downstream transactional side effects.
- **Adversarial Mechanism**: A load generator pumps 25,000 requests/second at a checkout endpoint and reports 0% network errors because the server rapidly returns HTTP 200 with an empty JSON body or cached error payload; downstream payment ledger rows were never created.
- **Enforcement & Diagnostic**: Every k6 load scenario must enforce semantic verification oracles checking response payload integrity (e.g., asserting `response.json().order_status === 'RESERVED'`) and verifying physical database state sampling. Assertion-free load scripts trigger diagnostic `ERR_ASSERTION_FREE_TEST_DETECTED` and are rejected during test validation.
- **Forbidden Output Behavior**: The load testing framework is strictly forbidden from logging a scenario as successful based solely on HTTP status codes without payload validation and database consistency checks.

#### 3. Reject Flaky External Dependency [ADV-ED-01]
- **Vulnerability**: Running load tests against staging environments that make un-throttled calls to live third-party partner sandboxes (such as payment gateways, credit check APIs, or SMS notification providers), resulting in external rate-limiting, variable public internet jitter, and false-positive test failures.
- **Adversarial Mechanism**: Executing a 28,000 req/sec load test against live external credit card tokenization sandboxes causes immediate HTTP 429 Too Many Requests rejections and IP blacklisting from the vendor, invalidating the test run and forcing engineers to disable performance gates.
- **Enforcement & Diagnostic**: All third-party external dependencies must be intercepted at the network boundary and substituted with high-throughput, low-latency WireMock or Envoy mock clusters configured with deterministic delay distributions and zero external egress. Live outbound WAN calls during load test execution trigger diagnostic `ERR_FLAKY_EXTERNAL_DEPENDENCY_BLOCKED`.
- **Forbidden Output Behavior**: The system is strictly forbidden from allowing un-mocked external partner API endpoints in automated load testing topologies.

---

### Invariants and Contracts

    Open-Loop Arrival Rate Mandate [INV-LOAD-01]
      Production qualification load tests must use open-loop arrival rate executors.
      Closed-loop virtual user models that artificially throttle request dispatch during server latency spikes are prohibited.

    Mutating Write Traffic Representation [INV-LOAD-02]
      Load testing suites must incorporate at least 30% state-mutating write transactions.
      Qualifying capacity based solely on cached read operations is barred from release sign-offs.

    Seventy-Five Percent Database Pool Ceiling [INV-LOAD-03]
      Database active connection pool saturation must not exceed 75% during peak 28,000 req/sec load.
      Breaching 75% connection saturation fails the capacity qualification gate automatically.

## Explicit Unknowns

- AWS Aurora PostgreSQL storage write IOPS burst throttling behavior during sustained 4-hour 20,000 TPS write plateaus (G-1).
- TLS session cache memory consumption on Envoy ingress proxies under 15,000 newly established connections per second (G-2).

## Traceability

| Claim | Classification | Source | Freshness |
|---|---|---|---|
| Peak 28,000 arrival requests/sec | provided | Black Friday capacity forecast | Current |
| 4-hour sustained 20,000 req/sec plateau | provided | Peak event operational specification | Current |
| Incident INC-3890 connection pool collapse | provided | Historical post-mortem | Historical |
| Latency budget p95 <= 45ms, p99 <= 85ms | provided | Core Banking SLA | Current |
| Open-loop Poisson arrival rate model | decided | David O'Reilly (Lead SRE Architect) | 2026-09-15 |
| 75% database connection pool ceiling | decided | Architectural invariant INV-LOAD-03 | 2026-09-15 |
| Reject pyramid by habit, assertion-free tests, and external flakes | decided | Task domain rules | 2026-09-15 |


## Verification

| Gate | Command | Exit | Evidence time |
|---|---|---:|---|
| Capacity Envelope Check | `python scripts/verify_load_capacity_envelope.py --service core-checkout-service` | 0 | 2026-09-16T10:14:00Z |
| Test Layer Conformance Gate | `bash scripts/check_test_layer_alignment.sh --target staging-cluster-e2e` | 0 | 2026-09-16T10:14:30Z |
| AST Assertion Linter Gate | `python scripts/lint_assertion_free_tests.py --target k6/scenarios` | 0 | 2026-09-16T10:15:05Z |
| Prometheus Oracle Telemetry Check | `python scripts/check_prometheus_load_oracle.py --query-file oracles/load-slo.promql` | 0 | 2026-09-16T10:15:40Z |
| Distributed Peak 28k Run | `k6 run --config k6/config.json k6/scenarios/black_friday_checkout.js` | 0 | 2026-09-16T11:02:15Z |


Reviewer self-check against load testing standards:
- **Workload Fidelity**: PASS. Open-loop arrival model with 70% read / 30% write realistic payload mix.
- **Scale Capacity**: PASS. 12-node k6 cluster sustains 28,000 req/sec across 4-hour soak tests.
- **Resource Protection**: PASS. Hard 75% database pool ceiling prevents connection exhaustion cascades.
- **Adversarial Defenses**: PASS. Explicit diagnostics for pyramid by habit, assertion-free tests, and external flakiness.
- **Markdown Hygiene**: PASS. Native Markdown syntax strictly adheres to `rule_markdown.md`.

## Open Decisions

- `DEC-LOAD-01`: David O'Reilly to determine whether automated capacity load tests should run as a weekly scheduled CI/CD performance gate in staging (Owner: David O'Reilly).

## Next steps

1. Marcus Vance provisions 12-node EKS load runner pool and installs k6 operator.
2. Platform team seeds 50,000 synthetic merchant and customer test accounts in Aurora staging tables.
3. Conduct staging dry run executing the 28,000 req/sec flash surge scenario to verify database connection pool headroom.
4. Integrate `verify_perf_gate.py` into deployment pipeline prior to Black Friday code freeze.