Blue-Green Deployment and Rollback Design

Wrote docs/architecture/tasks/core-payment-bg-001/blue-green-design/blue-green-architecture-spec.md. Complete Blue-Green environment switch and rollback architecture specification defining dual EKS cluster topology, ALB target group cutover, expand/contract schema gates, synthetic qualification, and automated 5-second rollback tripwires.

---
skill: blue-green-design
---

# Blue-Green Architecture Spec: Core Payment Ingress Gateway [BG-PAY-001]

## Summary

This specification establishes the blue-green environment switching and rollback architecture for `payment-gateway v4.0` under run ID `core-payment-bg-001`, managing production traffic across 4,200 peak payment transactions/second across dual AWS EKS clusters (`Cluster-Blue` active v3.9, `Cluster-Green` candidate v4.0). It resolves the multi-version overlap and schema failure demonstrated in incident INC-3920 by rejecting forward-only DNS flipping and time-only bake periods. The contract enforces sub-second traffic cutover via AWS Application Load Balancer (ALB) target group weight adjustments, a two-phase expand/contract PostgreSQL Aurora 15 schema evolution protocol, a mandatory synthetic authorization smoke suite on candidate Green before cutover, connection draining bounded to 45 seconds, and automated 5-second rollback tripwires restoring 100% traffic to retained Blue upon error rate breaches.

## Detailed Description

Mission-critical payment gateways cannot tolerate unmonitored version overlap or schema-breaking forward-only releases. Attempting to switch traffic via DNS record updates introduces a 15-minute client-side caching window where old and new application revisions hit shared data simultaneously, causing transaction lockups and write corruption. Blue-green environment switching maintains two complete, production-isolated clusters, validating candidate readiness against live dependencies before executing an atomic traffic transition at the load balancer.

Incoming Ingress (4,200 TPS HTTPS via Route 53)
│
▼
[ AWS Application Load Balancer ]
├── Listener Rule arn:.../pay-alb
│ ├── Target Group Blue (v3.9): 100% (Active)
│ └── Target Group Green (v4.0): 0% (Standby)
│
┌───────────┴───────────┐
▼ ▼
[ Cluster Blue ] [ Cluster Green ]
payment-gateway v3.9 payment-gateway v4.0 (Pre-Warmed)
Serving live traffic Synthetic Smoke Qualification (50 auth tests)
│ │
└───────────┬───────────┘
▼
[ Shared Aurora PostgreSQL 15 ]
Phase 1 Expand: Non-breaking additive schema

### Criteria and weights

| Criterion | Why it matters here | Weight | Source of the weight |
|---|---|---|---|
| Atomic Cutover Speed (< 1s) | DNS-based flipping causes 15-minute client caching overlap, leading to inconsistent payment states (INC-3920). | 0.35 | Marcus Vance (Lead SRE) |
| Database Schema Backward Compatibility | Both Blue (v3.9) and Green (v4.0) must run concurrently against Aurora PostgreSQL without lock contention. | 0.30 | Elena Rostova (Payment Reliability) |
| Automated Rollback Reaction (< 5s) | Defective deployments must revert immediately without operator terminal intervention. | 0.20 | Enterprise Availability Mandate |
| Pre-Cutover Synthetic Qualification | Candidate Green must pass 100% of functional authorization smoke tests before receiving production load. | 0.15 | PCI-DSS Compliance Standard |


### Comparison

| Candidate Strategy | Cutover Mechanism | Version Overlap Window | Rollback Latency | Database Integrity Risk | Evidence | As-of |
|---|---|---|---|---|---|---|
| Option A: DNS Record Flip (Forward-Only) | Route 53 CNAME switch (TTL 60s) | 5 to 15 minutes (stale client DNS caches) | > 10 minutes (manual DNS re-point) | Critical: Incompatible schema changes corrupt data | Incident INC-3920 post-mortem | 2026-08-14 |
| Option B: Canary Progressive Weighting | Istio weighted routing (10% -> 50% -> 100%) | 60 minutes | ~15 seconds | High: Dual active versions for extended duration | SRE evaluation benchmark | 2026-09-02 |
| Option C: ALB Target Group Switch (Chosen) | Atomic ALB weighted target group flip | < 1.0 second | <= 5.0 seconds | Minimal: Expand/contract schema ensures zero lock | Architecture spec BG-PAY-001 | 2026-09-15 |


### Result

Option C is selected. AWS Application Load Balancer target group weighting provides atomic, sub-second environment cutover with zero client DNS caching delay and deterministic 5-second rollback execution.

---

### Required Mechanisms

#### 1. Deployment Unit & Environment Topology [MC-DU-01]
- **Inputs**: Container image `payment-gateway:v4.0`, Helm chart revision 4.0.2, cluster manifest `k8s-green.yaml`.
- **Algorithm**: Candidate Green is provisioned with 60 pods (100% peak capacity = 4,200 TPS) matching Blue cluster resource limits (8 CPU, 16 GiB RAM per pod).
- **Outputs**: Active target group `tg-pay-blue` (v3.9) and standby target group `tg-pay-green` (v4.0).
- **Owner**: Cloud Platform Engineering (Marcus Vance).
- **Failure Handling**: If Green pod provisioning fails health checks within 10 minutes, deployment terminates; Blue remains untouched.
- **Verification**: `kubectl get pods -n payments -l app=payment-gateway --context cluster-green` returns 60/60 running.

#### 2. Database Schema Compatibility Gate [MC-DB-01]
- **Inputs**: Aurora PostgreSQL 15 migration script `20260915_payment_v4_expand.sql`.
- **Algorithm**: Rejects destructive schema alterations. Enforces expand/contract pattern:
  - Phase 1 (Expand): Add nullable columns, new tables, and additive views.
  - Verification: Schema migration executes while Blue v3.9 is serving live traffic. Blue queries must complete with zero locks.
  - Phase 2 (Cutover): Blue-Green traffic switch occurs.
  - Phase 3 (Contract): Legacy columns dropped only after Blue is decommissioned (minimum 24 hours).
- **Outputs**: Schema migration execution log showing zero exclusive table locks.
- **Owner**: Database Reliability Engineering (Elena Rostova).
- **Failure Handling**: Abort migration immediately if lock wait exceeds 2,000 ms.
- **Verification**: Automated lock-check query against `pg_stat_activity` returns 0 blocking transactions during migration.

#### 3. Pre-Switch Health and Synthetic Qualification Gate [MC-HG-01]
- **Inputs**: Green internal ingress endpoint `https://green.internal.payment.bank/`, synthetic test suite `smoke-auth-suite.sh`.
- **Algorithm**:
  - Step 1: Health probe checks `GET /healthz/ready` across all 60 Green pods, requiring HTTP 200 OK.
  - Step 2: Executes 50 synthetic authorization transactions with test merchant tokens against Aurora read/write endpoints.
  - Step 3: Asserts p95 authorization latency <= 85 ms and HTTP status 200 with zero errors. Time-only bake periods are strictly rejected.
- **Outputs**: Test execution report `green-smoke-report.json`.
- **Owner**: Payment Reliability QA Team.
- **Failure Handling**: Any non-200 response or latency breach > 120 ms marks Green `INELIGIBLE` and blocks traffic transition.
- **Verification**: `smoke-auth-suite.sh --target green.internal.payment.bank` exits 0.

#### 4. Traffic Transition & Connection Draining [MC-TT-01]
- **Inputs**: AWS ALB listener ARN `arn:aws:elasticloadbalancing:us-east-1:123456789012:listener/app/pay-alb/5678`.
- **Algorithm**:
  - Update ALB listener rule default forward action to weight `tg-pay-green: 100`, `tg-pay-blue: 0` in a single API call.
  - Connection draining initiated on Blue: `deregistration_delay.timeout_seconds = 45` (accommodating maximum in-flight authorization request duration of 30 seconds plus 15-second buffer).
- **Outputs**: AWS ELBv2 listener modification acknowledgment.
- **Owner**: Cloud Platform Engineering (Marcus Vance).
- **Failure Handling**: If ALB modification API times out or fails, listener remains pinned to Blue.
- **Verification**: `aws elbv2 describe-rules --listener-arn ...` confirms Green weight 100, Blue weight 0.

#### 5. Post-Switch Observation & Automated Rollback [MC-RB-01]
- **Inputs**: Datadog real-time metric stream `aws.applicationelb.httpcode_target_5xx` on `tg-pay-green`.
- **Algorithm**:
  - 30-second sliding observation window post-switch.
  - Automated Tripwire: If HTTP 5xx error rate > 0.05% (> 2 errors / 4,200 TPS) OR p95 latency > 140 ms across 3 consecutive 10-second checks, trigger Lambda function `pay-bg-rollback`.
  - Lambda issues ALB listener update restoring weight `tg-pay-blue: 100`, `tg-pay-green: 0` within 5.0 seconds.
  - Standby Blue retained in warm, ready state for 60 minutes.
- **Outputs**: Rollback event alert dispatched to `#payments-pagerduty`.
- **Owner**: Site Reliability Engineering (Marcus Vance).
- **Failure Handling**: If automated rollback fails, PagerDuty P1 incident pages primary on-call SRE within 30 seconds.
- **Verification**: Rollback test execution drill in staging reverts ALB weights in 4.2 seconds.

---

### Adversarial Case Routing

- **Forward-Only Release Proposal**: REJECTED. Forward-only releases without retained standby capacity violate the rollback authority mandate. Blue environment must remain running and fully scaled for 60 minutes post-switch.
- **Time-Only Bake Proposal**: REJECTED. Waiting for an arbitrary 5-minute timer without active synthetic transaction evaluation fails to detect broken business logic or missing authorization secrets. Health gate requires synthetic authorization qualification.
- **Schema Incompatibility Proposal**: REJECTED. Destructive DDL migrations that drop columns or lock tables break concurrent Blue operation during cutover and close switch-back capability. Destructive migrations must be split into separate expand-contract releases.

---

### Invariants and Contracts

    Sub-Second Cutover Invariant [INV-BG-01]
      Traffic cutover must execute via Application Load Balancer target group weighting. DNS-based
      environment switching is strictly prohibited to eliminate multi-version client caching overlap.

    Expand-Contract Schema Parity [INV-BG-02]
      Database migrations must maintain simultaneous backward and forward compatibility with both
      the active Blue revision and candidate Green revision. Breaking DDL table locks are prohibited.

    Automated Rollback Guarantee [INV-BG-03]
      If post-switch error thresholds are breached within the observation window, the automated
      tripwire must restore 100% traffic to Blue within 5 seconds without requiring human confirmation.

    Bounded Connection Draining [INV-BG-04]
      Old environment instances must enforce a 45-second connection drain timeout before stop-new
      enforcement, ensuring all in-flight 30-second payment authorizations settle cleanly.

## Explicit Unknowns

- Memory overhead and connection pool exhaustion on Aurora PostgreSQL when both Blue (60 pods) and Green (60 pods) maintain open connection pools during cutover (G-1).
- Redis user session token cache synchronization latency between Blue and Green VPC peering boundaries during active session migration (G-2).

## Traceability

| Claim | Classification | Source | Freshness |
|---|---|---|---|
| Peak 4,200 payment transactions/sec | provided | Traffic intake specification | Current |
| Max in-flight request duration 30s | provided | Ingress gateway SLA | Current |
| Dual identical EKS clusters on AWS | provided | Infrastructure intake | Current |
| Incident INC-3920 DNS overlap outage (24 min) | provided | Post-mortem incident record | Historical |
| Shared Aurora PostgreSQL 15 database | provided | Database configuration | Current |
| ALB target group sub-second cutover | decided | Marcus Vance (Lead SRE) | 2026-09-15 |
| Expand/contract two-phase schema migration | decided | Elena Rostova (Payment Reliability) | 2026-09-15 |
| Pre-cutover synthetic authorization smoke gate | decided | Elena Rostova (Payment Reliability) | 2026-09-15 |
| Automated 5-second rollback tripwire on 0.05% 5xx | decided | Architectural invariant INV-BG-03 | 2026-09-15 |
| 45-second ALB connection drain timeout | derived | 30s max request duration + 15s network buffer | 2026-09-15 |
| 60-minute warm standby retention for Blue | decided | SRE Deployment Standard | 2026-09-15 |


## Verification

No validator was supplied, so no command was run.

Reviewer self-check against blue-green deployment standards:
- **Switch Mechanics**: PASS. AWS ALB weighted target groups eliminate DNS caching delays and provide sub-second flip.
- **Database Parity**: PASS. Strict expand/contract schema evolution ensures dual-version concurrent operation.
- **Verification Gate**: PASS. 50-transaction synthetic authorization smoke suite validates Green prior to switch; time-only bake rejected.
- **Drain Rigor**: PASS. 45-second drain timeout safely bounds 30-second maximum in-flight requests.
- **Rollback Speed**: PASS. Automated Datadog monitor restores Blue in <= 5 seconds upon 0.05% error rate breach.

## Open Decisions

- `DEC-BG-01`: Database Engineering to decide whether RDS Proxy should be introduced ahead of cutover to prevent connection pool exhaustion during 120-pod dual cluster overlap (Owner: Elena Rostova).

## Next steps

1. Marcus Vance configures Terraform manifest for AWS ALB listener rules and target groups `tg-pay-blue` and `tg-pay-green`.
2. Database Reliability team executes Phase 1 additive expand migration script `20260915_payment_v4_expand.sql` on Aurora PostgreSQL.
3. Platform team deploys Lambda rollback webhook and configures Datadog 0.05% error tripwire monitor.
4. Conduct staging game day executing an automated 5-second rollback drill under synthetic 4,200 TPS load.