Repair Secret Rotation Outages

Project Detected
NestJS TypeScript SaaS running on Kubernetes with PostgreSQL, TypeORM, Vault-backed database credentials, BullMQ workers, and multiple API replicas

---

Rotation Failure Found

• Credential distribution and application activation are treated as the same event, while existing TypeORM pools continue using old authenticated connections and replacement connections may load a different credential generation
• Secret type: Database credential
• Secret identity: PostgreSQL application principal
• Current version: db-credential-v17
• Next version: db-credential-v18
• Failure result: Partial database authentication outage appears gradually as old pooled connections are replaced

---

Secret Contract

• Authoritative source: Vault database credential path used by the Kubernetes workloads
• Issuer: Vault-backed database credential workflow
• Active version: db-credential-v17 before cutover
• Accepted versions: v17 and v18 during the disposable dual-principal test
• Overlap support: Full in the tested dual-principal fixture; live Vault/PostgreSQL policy remains subject to staging verification
• Reload mechanism: Controlled TypeORM datasource replacement plus rolling workload restart for startup-only consumers
• Revocation authority: Existing credential-revocation workflow in the Vault/database integration

---

Consumer Inventory

• Producers: 6 API replicas opening PostgreSQL connections
• Validators: PostgreSQL authentication layer
• Workers and scheduled jobs: 8 BullMQ workers, 2 scheduled maintenance jobs, and 1 migration runner
• Connection pools or cached clients: TypeORM pools in every API and worker process
• Standby / DR consumers: 2 standby deployment manifests identified but not executed in the isolated test
• Consumers still on old version: 0 after the completed test rollout

---

Rotation Plan

• Prepare: Issue v18 with equivalent required database privileges while v17 remains valid
• Distribute: Publish v18 through the test Vault path and make workloads aware of the immutable credential version
• Overlap: Verify new v18 connections while existing v17 pools remain available
• Cutover: Replace datasource clients and roll startup-only consumers to v18 in bounded batches
• Verification gate: Require every API, worker, scheduled-job fixture, and migration connection to establish a fresh v18 database connection
• Old-version revocation gate: Zero required consumers report v17 and fresh v18 connections pass after pool churn
• Rollback boundary: v17 remains available until the revocation gate passes

---

Repair Applied

• Secret version handling: Added immutable credential-generation metadata to the Vault loader and safe version reporting
• Consumer reload behavior: Added controlled datasource replacement for reload-capable services and rolling restart hooks for startup-only workers
• Dual-version or overlap handling: Added a staged dual-principal test path rather than replacing the active password in one step
• Cutover and health gates: Added fresh-connection readiness checks instead of relying on already-authenticated pooled connections
• Revocation and rollback protection: Old credential cannot enter revocation state until every required consumer generation is verified

---

Files Modified

• src/config/databaseSecret.loader.ts
• src/database/datasourceManager.ts
• src/database/databaseCredentialHealth.ts
• src/workers/workerBootstrap.ts
• src/rotation/databaseRotation.service.ts
• k8s/api-deployment.yaml
• k8s/worker-deployment.yaml
• test/database-secret-rotation.spec.ts

---

Commands Executed

• npm run typecheck
• npm test -- database-secret-rotation.spec.ts
• npm run build

---

Overlap Test

• Consumers tested: 17 API, worker, scheduled-job, and migration-runner fixtures
• Consumers accepting old version: 17 before migration
• Consumers accepting new version: 17 after v18 distribution
• Authentication failures during overlap: 0

---

Cutover Test

• Producers switched: 17
• Consumers on new version: 17
• New authentication failures: 0
• Old-version operations after cutover: 0 after forced connection-pool churn

---

Revocation Test

• Old version revoked: Yes in the disposable database fixture
• New version remained valid: Yes
• Consumers still requiring old version: 0
• Outages after revocation: 0

---

Failure Recovery Test

• Failure point: One worker rejected v18 during the readiness stage before fleet cutover
• Last known good version: db-credential-v17
• Rollback result: Rotation remained in overlap state, v17 stayed active, the faulty worker was restored, and revocation was blocked
• Consumers left without usable credential: 0

---

Validation Result
Type checking, 34 secret-loading, fresh-connection, pool-churn, rolling-cutover, worker, scheduled-job, rollback, and revocation tests, and the production build completed successfully using isolated PostgreSQL, mocked Vault responses, and Kubernetes deployment fixtures.

---

Remaining Risks

• Live Vault lease semantics, PostgreSQL credential overlap, backup tooling, and production Kubernetes rollout timing require staging verification.

---

Important Notes
No production Vault credential or PostgreSQL user was changed or revoked. Existing pooled connections were not treated as proof that the new credential worked; every required path created fresh authenticated connections before the old generation became revocation-eligible.