Repair Secret Rotation Outages

Project Detected
NestJS TypeScript SaaS running on Kubernetes with PostgreSQL, TypeORM, Vault-backed database credentials, BullMQ workers, and multiple API replicas

Rotation Failure Found

• Credential distribution and application activation are treated as the same event, while existing TypeORM pools continue using old authenticated connections and replacement connections may load a different credential generation
• Secret type: Database credential
• Secret identity: PostgreSQL application principal
• Current version: db-credential-v17
• Next version: db-credential-v18
• Failure result: Partial database authentication outage appears gradually as old pooled connections are replaced

Secret Contract

• Authoritative source: Vault database credential path used by the Kubernetes workloads
• Issuer: Vault-backed database credential workflow
• Active version: db-credential-v17 before cutover
• Accepted versions: v17 and v18 during the disposable dual-principal test
• Overlap support: Full in the tested dual-principal fixture; live Vault/PostgreSQL policy remains subject to staging verification
• Reload mechanism: Controlled TypeORM datasource replacement plus rolling workload restart for startup-only consumers
• Revocation authority: Existing credential-revocation workflow in the Vault/database integration

Consumer Inventory

• Producers: 6 API replicas opening PostgreSQL connections
• Validators: PostgreSQL authentication layer
• Workers and scheduled jobs: 8 BullMQ workers, 2 scheduled maintenance jobs, and 1 migration runner
• Connection pools or cached clients: TypeORM pools in every API and worker process
• Standby / DR consumers: 2 standby deployment manifests identified but not executed in the isolated test
• Consumers still on old version: 0 after the completed test rollout

Rotation Plan

• Prepare: Issue v18 with equivalent required database privileges while v17 remains valid
• Distribute: Publish v18 through the test Vault path and make workloads aware of the immutable credential version
• Overlap: Verify new v18 connections while existing v17 pools remain available
• Cutover: Replace datasource clients and roll startup-only consumers to v18 in bounded batches
• Verification gate: Require every API, worker, scheduled-job fixture, and migration connection to establish a fresh v18 database connection
• Old-version revocation gate: Zero required consumers report v17 and fresh v18 connections pass after pool churn
• Rollback boundary: v17 remains available until the revocation gate passes

Repair Applied

• Secret version handling: Added immutable credential-generation metadata to the Vault loader and safe version reporting
• Consumer reload behavior: Added controlled datasource replacement for reload-capable services and rolling restart hooks for startup-only workers
• Dual-version or overlap handling: Added a staged dual-principal test path rather than replacing the active password in one step
• Cutover and health gates: Added fresh-connection readiness checks instead of relying on already-authenticated pooled connections
• Revocation and rollback protection: Old credential cannot enter revocation state until every required consumer generation is verified

Files Modified

• src/config/databaseSecret.loader.ts
• src/database/datasourceManager.ts
• src/database/databaseCredentialHealth.ts
• src/workers/workerBootstrap.ts
• src/rotation/databaseRotation.service.ts
• k8s/api-deployment.yaml
• k8s/worker-deployment.yaml
• test/database-secret-rotation.spec.ts

Commands Executed

• npm run typecheck
• npm test -- database-secret-rotation.spec.ts
• npm run build

Overlap Test

• Consumers tested: 17 API, worker, scheduled-job, and migration-runner fixtures
• Consumers accepting old version: 17 before migration
• Consumers accepting new version: 17 after v18 distribution
• Authentication failures during overlap: 0

Cutover Test

• Producers switched: 17
• Consumers on new version: 17
• New authentication failures: 0
• Old-version operations after cutover: 0 after forced connection-pool churn

Revocation Test

• Old version revoked: Yes in the disposable database fixture
• New version remained valid: Yes
• Consumers still requiring old version: 0
• Outages after revocation: 0

Failure Recovery Test

• Failure point: One worker rejected v18 during the readiness stage before fleet cutover
• Last known good version: db-credential-v17
• Rollback result: Rotation remained in overlap state, v17 stayed active, the faulty worker was restored, and revocation was blocked
• Consumers left without usable credential: 0

Validation Result
Type checking, 34 secret-loading, fresh-connection, pool-churn, rolling-cutover, worker, scheduled-job, rollback, and revocation tests, and the production build completed successfully using isolated PostgreSQL, mocked Vault responses, and Kubernetes deployment fixtures.

Remaining Risks

• Live Vault lease semantics, PostgreSQL credential overlap, backup tooling, and production Kubernetes rollout timing require staging verification.

Important Notes
No production Vault credential or PostgreSQL user was changed or revoked. Existing pooled connections were not treated as proof that the new credential worked; every required path created fresh authenticated connections before the old generation became revocation-eligible.