Context: GitLab CI → deploy payments-api to staging Kubernetes (EKS). Symptom: deploy:staging job fails; pods never Ready. Production not involved.

kubectl get pods -n payments-staging: payments-api-7f8c9d4b6-xk2m1 0/1 CrashLoopBackOff 6 11m payments-api-5a1b2c3d4-m8n7w 1/1 Running 0 2d

kubectl describe (excerpt): Readiness probe failed: HTTP probe failed with statuscode: 500 Back-off restarting failed container payments-api

kubectl logs --previous (excerpt): ERROR config validation failed: missing required env STRIPE_WEBHOOK_SECRET FATAL refusing to start with invalid config process exit code=1

GitLab CI / Helm (tail): helm upgrade --install payments-api ... --set image.tag=2.14.0 --wait --timeout 5m Error: UPGRADE FAILED: context deadline exceeded ERROR: Job failed: exit code 1

Pipeline Failure RCA Report

1. Executive Summary

The staging Helm rollout for payments-api:2.14.0 failed because new pods enter CrashLoopBackOff: process exits with code 1 after config validation fails on missing STRIPE_WEBHOOK_SECRET, readiness probes return HTTP 500, and helm upgrade --wait times out. Old pods remain healthy; production was not touched.

2. Root Cause Analysis (Evidence-Backed)

| Failure Domain | Specific Error | Log Evidence | | Infrastructure / Config | Missing required env | missing required env STRIPE_WEBHOOK_SECRET | | Runtime | Exit code 1 | refusing to start with invalid config | | Kubernetes Health | Readiness 500 | HTTP probe failed with statuscode: 500 | | CI/CD Deploy | Helm wait timeout | UPGRADE FAILED: context deadline exceeded |

3. Immediate Recovery Plan (human-run only)

1. Inspect Secret + Deployment env refs in payments-staging
2. Apply STRIPE_WEBHOOK_SECRET from vault (never commit secrets)
3. helm upgrade … --wait; verify READY 1/1 and /health 200
4. If blocked: helm rollback payments-api -n payments-staging

4. Long-Term Prevention

• CI pre-check for required secrets in target namespace
• Helm lint fails when secretKeyRefs unset
• Boot integration test with minimal env fixture

5. Blast Radius & Rollback

• Data impact: None (config at startup)
• Downtime risk: Staging rollout stuck; old pod still Running
• Rollback: helm rollback (staging only) — no namespace delete

Incident Management JSON

{ "incident_type": "Kubernetes Rollout Failure", "severity": "Medium (Staging)", "failure_domain": "Infrastructure/Config", "root_cause_summary": "payments-api:2.14.0 CrashLoopBackOff because STRIPE_WEBHOOK_SECRET is missing; readiness 500 and helm --wait timeout.", "immediate_fix_commands": ["kubectl get secret -n payments-staging", "helm rollback payments-api -n payments-staging", "..."], "prevention_tags": ["kubernetes", "helm", "secrets", "crashloopbackoff", "gitlab-ci"] }