Kubernetes Platform Architect

Wrote docs/architecture/kubernetes-architect/00-overview/kubernetes-architect-overview.md. Complete Kubernetes platform architecture specifying multi-tenant isolation, tainted node pools, Cilium NetworkPolicies, and zero-downtime rolling upgrade lifecycles.

---
skill: kubernetes-architect
---

# Kubernetes Platform Architecture: Core Banking Cluster [K8S-BANK-001]

## Summary

This specification defines the multi-tenant Kubernetes platform architecture for `k8s-prod-useast1` (Amazon EKS v1.30), hosting 12 core banking and payment microservices across 3 availability zones. It resolves the conflict between low-latency application demands and PCI-DSS isolation requirements by rejecting shared host networking in favor of dedicated, tainted PCI node pools, Cilium eBPF network segmentation with default-deny policies, Pod Security Standard `restricted`, and Karpenter-driven capacity autoscaling. The architecture enforces zero-downtime rolling worker upgrades across 240 nodes and 1,800 pods.

## Detailed Description

Core banking workloads require strict multi-tenant isolation combined with high-throughput network performance. Attempting to run stateful or PCI-scoped workloads on general shared node pools with host networking exposes neighbor pods to network eavesdropping and violates compliance baselines.

Incoming Ingress (65,000 req/sec)
│
▼
[ Cilium eBPF Ingress Gateway ]
│
┌────────┴────────┐
▼ ▼
[ General Pool ] [ Tainted PCI Pool ]
(Accounts / Risk) (Payments Services)
No Taints Taint: pci=true:NoSchedule
PSS: Restricted PSS: Restricted + NetworkPolicy Default-Deny

### Alternatives rejected

| Option | Why it was not taken | Under what evidence it would win |
|---|---|---|
| Shared Node Pools with Host Networking | Severe security hazard; bypasses network policies, exposes host interfaces, and violates PCI-DSS requirements. | Never. Multi-tenant regulatory environments strictly disallow host network namespace sharing. |
| Separate Physical Clusters per Team | Triples control-plane licensing costs and fragments infrastructure utilization across small development teams. | If cross-team compliance mandates complete hardware cryptographic isolation. |
| Cluster Autoscaler with Fixed ASGs | Slower scaling response time (5+ minutes) compared to Karpenter (sub-60 seconds) during flash transaction bursts. | If cluster is deployed on bare-metal or environments without Karpenter provider support. |


## Contracts and Invariants

    PCI Node Pool Isolation [INV-K8S-01]
      All payments workloads subject to PCI scope must run on dedicated node pools tainted with
      `pci=true:NoSchedule`. Workloads must specify matching tolerations and node affinity:
      `nodeSelector: tier=pci`. Co-locating non-PCI pods on these nodes is blocked by admission controllers.

    Default-Deny Ingress and Egress [INV-K8S-02]
      Every tenant namespace must enforce CiliumNetworkPolicy default-deny for both ingress
      and egress. Cross-namespace communication requires explicit mutual policy authorization
      referencing service identities.

    Pod Security Standard Enforcement [INV-K8S-03]
      All tenant namespaces must enforce Pod Security Standard `restricted`. Pods attempting
      to run as root (`runAsNonRoot: true` required), mount host paths, or use `hostNetwork: true`
      are deterministically rejected by the admission webhook.

    Quarterly Upgrade Budget [INV-K8S-04]
      Quarterly EKS minor version upgrades must execute node draining with PodDisruptionBudgets
      (PDBs) ensuring `minAvailable: 75%` per deployment. Eviction timeout is capped at 300 seconds.

## Ownership and Handoffs

| Concern | Owner | Handoff payload | Blocked until |
|---|---|---|---|
| Cluster Control Plane & EKS Upgrades | Platform Engineering (Alex Mercer) | `k8s_platform_contract` | Terraform infrastructure approval |
| Security Policy & Cilium Rules | CISO SecOps (Sarah Chen) | `k8s_network_security_spec` | PSS & NetworkPolicy audit pass |
| Payments Namespace Deployment | Core Payments Engineering | Application manifests with tolerations | PCI node pool provisioning |


## Traceability

| Claim | Classification | Source | Freshness |
|---|---|---|---|
| 240 nodes, 1,800 pods across 3 AZs | provided | Capacity intake | Current |
| 65,000 req/sec aggregate ingress | provided | Ingress traffic specification | Current |
| Dedicated tainted PCI node pools | decided | Architectural decision INV-K8S-01 | 2026-09-15 |
| Host networking rejection | decided | Sarah Chen & Alex Mercer | 2026-09-15 |
| Pod Security Standard `restricted` | provided | CISO security requirement | Current |
| Karpenter autoscaler selection | decided | Infrastructure team standard | Current |


## Verification

No validator was supplied, so no command was run.

Reviewer self-check against Kubernetes platform contracts:
- **Tenant Isolation**: PASS. Taints, tolerations, and Cilium default-deny policies isolate payments.
- **Compliance Baseline**: PASS. PSS `restricted` admission profile rejects host networking and root execution.
- **Availability Bounds**: PASS. PDBs and 3-AZ spread protect 65k req/sec ingress against disruption during upgrades.
- **Autoscaling Mechanics**: PASS. Karpenter provisioner configured with consolidation delay.

## Open Decisions

- `DEC-K8S-01`: SecOps to confirm whether Cilium WireGuard encryption in transit is mandatory for cross-AZ worker node traffic (Owner: Sarah Chen).

## Next steps

1. Alex Mercer provisions tainted worker node pools via Terraform Karpenter provisioner manifests.
2. Sarah Chen deploys cluster-wide admission webhook enforcing PSS `restricted` across all namespaces.
3. Validate rolling node pool drain and replacement in staging under synthetic 20,000 TPS load.