Pipeline Failure RCA

Pipeline Failure RCA Report

1. Executive Summary

The staging Helm rollout for payments-api:2.14.0 failed because new pods enter
CrashLoopBackOff: process exits with code 1 after config validation fails on
missing STRIPE_WEBHOOK_SECRET, readiness probes return HTTP 500, and
helm upgrade --wait times out. Old pods remain healthy; production was not touched.

2. Root Cause Analysis (Evidence-Backed)

| Failure Domain | Specific Error | Log Evidence |
| Infrastructure / Config | Missing required env | missing required env STRIPE_WEBHOOK_SECRET |
| Runtime | Exit code 1 | refusing to start with invalid config |
| Kubernetes Health | Readiness 500 | HTTP probe failed with statuscode: 500 |
| CI/CD Deploy | Helm wait timeout | UPGRADE FAILED: context deadline exceeded |

3. Immediate Recovery Plan (human-run only)

1. Inspect Secret + Deployment env refs in payments-staging
2. Apply STRIPE_WEBHOOK_SECRET from vault (never commit secrets)
3. helm upgrade … --wait; verify READY 1/1 and /health 200
4. If blocked: helm rollback payments-api -n payments-staging

4. Long-Term Prevention

• CI pre-check for required secrets in target namespace
• Helm lint fails when secretKeyRefs unset
• Boot integration test with minimal env fixture

5. Blast Radius & Rollback

• Data impact: None (config at startup)
• Downtime risk: Staging rollout stuck; old pod still Running
• Rollback: helm rollback (staging only) — no namespace delete

Incident Management JSON

{
  "incident_type": "Kubernetes Rollout Failure",
  "severity": "Medium (Staging)",
  "failure_domain": "Infrastructure/Config",
  "root_cause_summary": "payments-api:2.14.0 CrashLoopBackOff because STRIPE_WEBHOOK_SECRET is missing; readiness 500 and helm --wait timeout.",

"immediate_fix_commands": ["kubectl get secret -n payments-staging", "helm rollback payments-api -n payments-staging", "..."],
"prevention_tags": ["kubernetes", "helm", "secrets", "crashloopbackoff", "gitlab-ci"]
}