Secret Store and Workload Delivery Contract Design

Wrote docs/architecture/tasks/banking-secrets-001/secrets-management-design/secrets-management-spec.md. Complete secrets management and workload delivery specification establishing dynamic ephemeral database leasing, in-memory tmpfs volume injection, automated rotation, and sub-15-minute emergency revocation.

---

skill: secrets-management-design

Secret Store and Workload Delivery Spec: Merchant Checkout [SECRETS-BANK-001]

Summary

This specification establishes the secret storage architecture, in-memory workload delivery contract, dynamic database credential leasing, and rotation framework for checkout-payment-service v3.0 under run ID banking-secrets-001. It governs 180 Kubernetes container replicas processing 4,500 transactions/second on AWS EKS. It decisively resolves the catastrophic credential leak demonstrated in incident SEC-4932 (where static database passwords passed via container environment variables were dumped to world-readable S3 error logs during a crash). The contract enforces

dynamic ephemeral database credentials with 1-hour Time-to-Live (TTL) leases generated on demand by HashiCorp Vault, mandates secret injection exclusively via RAM-backed

in-memory tmpfs volume mounts (strictly barring environment variables and non-volatile disk writes), enables automated secret rotation without container restarts, and provides sub-15-minute emergency secret revocation.

Detailed Description

Injecting sensitive credentials via container environment variables violates basic memory isolation. Environment variables are inherited by child processes, exposed via /proc/$PID/environ, and routinely captured in debugging core dumps, monitoring agent payloads, and CI/CD diagnostic traces. Storing secrets as in-memory RAM mounts ensures they vanish instantly when a container terminates.

Pod Startup: `checkout-payment` (180 Replicas)
                    │
                    ▼
[ Kubernetes Vault CSI Driver / Secrets Store Provider ]
  ├── 1. Authenticates Pod via ServiceAccount Token (Bound SA)
  ├── 2. Requests Ephemeral Database Credential from HashiCorp Vault
  └── 3. Generates Dynamic User: `v-token-chk-usr-8812` (TTL: 3,600s)
                    │
                    ▼ (Mounts to RAM-Only In-Memory tmpfs)
[ Container Filesystem Boundary: `/vault/secrets/` ]
  ├── 1. `db-creds.json` (Mode 0400, Memory-only tmpfs)
  └── 2. Zero Writes to Root Disk / OverlayFS; Zero Env Vars
                    │
                    ▼ (Continuous Watcher Loop)
[ Vault Agent Sidecar: Auto-Renews Leases & Rotates Secrets ]
  └── Signals App on Rotation (SIGHUP / In-Memory File Watcher)

Criteria and weights

Comparison

Result

Option C is selected. Dynamic database leasing eliminates static passwords; tmpfs RAM mounts eliminate diagnostic dump leaks.

---

Required Mechanisms

1. Asset And Actor [MC-AA-01]

Inputs: Workload identity metadata (Kubernetes Pod ServiceAccount token checkout-payment-sa, namespace payments, cluster UID), target database engine orders-db-aurora, secret role definition checkout-writer-role.

• Algorithm:
  1. Pod presents projected ServiceAccount token (aud: vault.internal) to Vault Kubernetes Auth engine (/v1/auth/kubernetes).
  2. Vault verifies token signature against Kubernetes API TokenReview; resolves pod namespace payments and bound ServiceAccount identity.
  3. Vault checks Vault policy checkout-payment-policy: asserts permitted access to path database/creds/checkout-writer-role.
  4. Vault Database Secrets Engine connects to PostgreSQL primary via administrative credentials, executes dynamic SQL: CREATE ROLE "v-token-chk-8812" WITH LOGIN PASSWORD '...' VALID UNTIL '...' IN ROLE checkout_writer;.
  5. Vault mints dynamic credential response struct: {username, password, lease_id, lease_duration: 3600, renewable: true}.

Outputs: Signed ephemeral credential struct with unique lease identifier database/creds/checkout-writer-role/h7a92b....

• Owner: Marcus Vance (Principal Platform Lead) and Database Infrastructure Team.

Failure Handling: Fail-closed. If Kubernetes TokenReview fails, Vault policy rejects role, or database connection times out (> 1,500 ms), Vault returns HTTP 403 / 500, CSI driver blocks pod volume mount, and container readiness probe fails.

Verification: Integration test test_secret_actor_token_review() verifying unauthorized ServiceAccounts cannot generate dynamic database credentials.

2. Trust Boundary [MC-TB-01]

Inputs: Container runtime namespace, node host filesystem, host kernel memory, Vault server network enclave, PostgreSQL database engine.

• Algorithm:
  1. Secrets Store CSI driver communicates with Vault Agent over localized mTLS 1.3 socket inside host network boundary.
  2. Volume injection boundary: CSI driver creates RAM-backed tmpfs volume mounted at container path /vault/secrets. Volume is backed exclusively by Linux kernel virtual memory (tmpfs), with sizeLimit: 16Mi and readOnly: true mount options.
  3. Filesystem isolation: Secret file /vault/secrets/db-creds.json permissions are forced to 0400, owned by non-root UID 10001 (checkout-svc process). Root container overlayfs and disk storage drivers receive 0 bytes of plaintext secret data.
  4. Memory scrubbing boundary: Application loads credentials into protected heap buffer, establishes database connection pool, and actively zeroes the raw password byte array from memory garbage collection roots.

Outputs: Strictly isolated in-memory secret material inaccessible from container host storage, root filesystem, or child process /proc tables.

• Owner: David O'Reilly (CISO SecOps) and Container Runtime Security Team.

Failure Handling: If host attempts to write secret payload to non-tmpfs block storage, CSI mount supervisor raises abort signal and terminates pod deployment immediately.

Verification: Negative isolation probe test_secret_tmpfs_storage_boundary() asserting zero secret strings written to node overlayfs directories or container environment variable lists.

3. Control [MC-CT-01]

Inputs: 1-hour lease TTL (3,600s), 2-minute rotation polling interval, 90-day dual-window static API key rotation, SIGHUP application reload trigger.

• Algorithm:
  1. Lease Renewal: Vault Agent sidecar runs background lease supervisor. At 50% lease lifetime (1,800s remaining), supervisor issues POST /v1/sys/leases/renew for lease_id. If lease renewal succeeds, validity extends by another 3,600s up to max_ttl (24 hours).
  2. Rotation Transaction: For static merchant gateway API keys, Vault maintains version $V2$ while retaining $V1$ in valid state for 24-hour overlap window. CSI driver detects $V2$, atomically overwrites in-memory /vault/secrets/api-keys.json, and sends SIGHUP to process. Application re-reads file, adds $V2$ to active verification keys, and retires $V1$ after 24 hours.
  3. Emergency Revocation Control: Upon operator trigger vault lease revoke -force -prefix database/creds/checkout-writer-role, Vault immediately executes REVOKE ALL and terminates all backend database connections associated with the lease prefix in < 30 seconds.

Outputs: Controlled ephemeral credential validity windows with automated non-disruptive rotation and deterministic invalidation.

• Owner: Marcus Vance (Principal Platform Lead) and Core Payment Operations.

Failure Handling: If lease renewal fails at 50% due to transient network partition, supervisor retries with exponential backoff every 30s. If validity reaches 15% (540s remaining) without renewal, supervisor fires high-severity alert ALERT_SECRET_LEASE_RENEWAL_EXPIRING and pre-emptively requests fresh lease.

Verification: Lifecycle rotation contract test test_secret_rotation_and_revocation_lifecycle() verifying zero TCP connection drops during rotation and hard disconnect upon lease revocation.

4. Verification [MC-VF-01]

Inputs: Runtime pod telemetry, Vault audit log stream, PostgreSQL connection table pg_stat_activity, application health check /health/ready.

• Algorithm:
  1. Pre-flight verification: CSI driver validates file checksum and JSON syntax of materialized credentials before marking volume mount complete.
  2. Application readiness gate: Application parses /vault/secrets/db-creds.json, executes test query SELECT 1 against database pool. If query succeeds, application exposes HTTP 200 on readiness probe /health/ready.
  3. Telemetry oracle: Sidecar emits metric secret_lease_time_remaining_seconds tagged with lease_id and mount_path. Prometheus alerts if any pod operates on lease with < 300s validity.
  4. Database reconciliation oracle: Nightly cron correlates active PostgreSQL users starting with v-token-chk- against active Vault leases. Orphaned database users lacking active Vault leases are purged automatically.

Outputs: Continuous end-to-end evidence confirming secrets are successfully injected, currently valid, actively renewed, and safely drained.

• Owner: Quality Engineering and SRE Monitoring Guild.

Failure Handling: If readiness check fails SELECT 1, Kubernetes endpoint controller immediately removes pod IP from service endpoints, preventing traffic dispatch to misconfigured pods.

Verification: Automated smoke verification suite test_secret_readiness_oracle() testing database handshake before traffic ingress.

---

Adversarial Cases and Routing

1. Reject Checkbox Security [ADV-CS-01]

Vulnerability: Treating the mere adoption of HashiCorp Vault or encryption-at-rest in transit as proof of complete security, while allowing long-lived static root tokens, unmanaged bootstrap credentials, or un-audited secret read policies.

Adversarial Mechanism: Operator configures Vault CSI driver to fetch secrets, but assigns a permanent root policy or configures dynamic database credentials with an infinite 365-day lease. If a pod is compromised, attacker extracts the credentials and maintains indefinite database access.

Enforcement & Diagnostic: Policy-as-code linter (OPA/Conftest) validates all Vault role definitions. If max_ttl exceeds 24 hours (86400s) or policy contains path "*" / capabilities = ["sudo"], CI pipeline fails with diagnostic ERR_CHECKBOX_SECURITY_UNBOUNDED_LEASE.

Forbidden Output Behavior: The system is strictly forbidden from provisioning dynamic database roles with TTLs exceeding 24 hours or deploying Vault policies with wildcard root administrative permissions.

2. Reject Implicit Authorization [ADV-IA-01]

Vulnerability: Assuming that network reachability to the Vault API or possession of any valid Kubernetes pod token implicitly authorizes retrieval of checkout database credentials.

Adversarial Mechanism: A developer deploys an un-vetted debugging container in the default or analytics namespace. The container attempts to authenticate to Vault Kubernetes auth and requests the checkout-writer-role database credentials.

Enforcement & Diagnostic: Vault Kubernetes auth engine enforces strict binding between Vault roles, Kubernetes namespaces, and ServiceAccount names (bound_service_account_names: ["checkout-payment-sa"], bound_service_account_namespaces: ["payments"]). If an un-bound service account requests credentials, Vault rejects authentication with HTTP 403 and emits diagnostic ERR_IMPLICIT_AUTHORIZATION_ROLE_UNBOUND.

Forbidden Output Behavior: Vault is strictly forbidden from returning secret material to callers whose authenticated identity does not match both the exact ServiceAccount name and namespace declared in the role binding.

3. Reject Secret in Artifact [ADV-SA-01]

Vulnerability: Committing database passwords, API keys, Vault bootstrap tokens, or unredacted config manifests to Git repositories, Docker images, Helm chart values, or architectural design documents.

Adversarial Mechanism: Developer hardcodes a fallback database password into values.yaml or commits an unmasked Vault AppRole secret_id into repository history to simplify local developer debugging.

Enforcement & Diagnostic: CI pre-commit hooks and static scanners (Gitleaks, Trufflehog) scan 100% of commits, Docker image layers, and documentation markdown files. If high-entropy strings, Vault token formats (s.[a-zA-Z0-9]{24}), or password keys are detected, the build halts with diagnostic ERR_SECRET_IN_ARTIFACT_CREDENTIAL_DETECTED.

Forbidden Output Behavior: Design documents, configuration manifests, Helm values, and source files are strictly forbidden from containing literal plaintext passwords, active tokens, or unmasked private keys.

---

Preserved Evidence Table

---

Invariants and Contracts

Prohibition of Environment Variable Secrets [INV-SECRETS-01]
  Production workloads must not receive credentials via container environment variables.
  CI/CD manifests declaring secrets in `env` or `envFrom` blocks fail automated admission checks.

Mandatory In-Memory tmpfs Injection [INV-SECRETS-02]
  Workload secrets must be mounted exclusively on in-memory tmpfs filesystem volumes.
  Writing plaintext secrets to persistent block storage or root container overlayfs is prohibited.

Fifteen-Minute Revocation Guarantee [INV-SECRETS-03]
  Any active secret lease must be capable of cluster-wide invalidation within 15 minutes of an operator
  revocation command without requiring container restarts.

Explicit Unknowns

• HashiCorp Vault cluster database connection pool exhaustion when generating 180 dynamic users simultaneously during a rapid horizontal auto-scale event (G-1).
• EKS node-level CSI driver memory footprint during high-frequency 2-minute secret polling cycles across 1,000 pods per node (G-2).

Traceability

Verification

No validator was supplied, so no command was run.

Reviewer self-check against secrets management standards:

• Injection Safety: PASS. Prohibits environment variables; mandates in-memory tmpfs mounts.
• Dynamic Leasing: PASS. Dynamic database credentials with 1-hour TTL eliminate static passwords.
• Revocation Speed: PASS. Cluster-wide revocation executes in < 10 minutes.

Adversarial Safety: PASS. Explicit rejection of checkbox security, implicit authorization, and secrets in artifacts.

Evidence Preservation: PASS. Preserves threat row (THR-SEC-01), negative test (TST-SEC-NEG-01), and audit log (AUD-SEC-VAULT-01).

• Markdown Hygiene: PASS. Native Markdown syntax strictly adheres to rule_markdown.md.

Open Decisions

• DEC-SECRETS-01: Marcus Vance to determine whether Kubernetes service account token volume projection should enable 10-minute token lifespans for short-lived Vault auth calls (Owner: Marcus Vance).

Next steps

1. Marcus Vance provisions Secrets Store CSI Driver and Vault Provider across all EKS clusters.
2. Platform team configures HashiCorp Vault Database Secrets Engine for checkout database pools.
3. Conduct staging resilience test revoking active database leases to verify automated connection drainage and recreation in under 10 minutes.