Kubernetes Ingress and Gateway API Design

Wrote docs/architecture/tasks/payment-ingress-001/ingress-design/ingress-architecture-spec.md. Complete Kubernetes ingress architecture specification establishing Gateway API routing, Envoy Gateway TLS 1.3 termination, PROXY Protocol v2 client IP preservation, and token-bucket rate limiting.

---
skill: ingress-design
---

# Ingress Architecture Spec: Core Payment Ingress Gateway [INGRESS-PAY-001]

## Summary

This specification establishes the Kubernetes north-south ingress traffic architecture, Gateway API routing policies, and edge security controls for `payment-ingress-gateway v2.0` under run ID `payment-ingress-001`, terminating 14,000 peak HTTPS requests/second across 4 AWS availability zones on EKS. It resolves the configuration chaos, security vulnerabilities, and vendor lock-in of legacy Kubernetes Ingress annotations by establishing the Kubernetes Gateway API (`GatewayClass`, `Gateway`, and `HTTPRoute`). The architecture enforces centralized TLS 1.3 termination via Envoy Gateway, automated certificate lifecycle management via cert-manager, end-to-end client IP preservation via AWS Network Load Balancer (NLB) PROXY Protocol v2, and token-bucket rate limiting (100 req/s burst 200) to protect backend services from layer-7 denial-of-service surges.

## Detailed Description

Legacy Kubernetes `Ingress` v1 objects rely on fragmented, vendor-specific controller annotations (`nginx.ingress.kubernetes.io/...`) that lack compile-time schema validation and blur the boundary between cluster operators and application squads. Furthermore, terminating TLS downstream at application pods prevents centralized WAF filtering, while terminating at external L4 load balancers without PROXY protocol masks client source IPs behind cloud NAT gateways.

Internet Clients (14,000 HTTPS req/sec)
│
▼
[ AWS Network Load Balancer (NLB) ] (Cross-Zone Enabled)
└── PROXY Protocol v2 Header Injected (Preserves True Client IP)
│
▼ (TCP / Port 443)
[ Kubernetes Gateway API: Envoy Gateway ]
├── 1. Central TLS 1.3 Termination (ECDHE-ECDSA-AES128-GCM-SHA256)
├── 2. cert-manager ACME / Let's Encrypt Automated Certificate Rotation
├── 3. Global Token-Bucket Rate Limiter (100 req/s, Burst: 200)
└── 4. HTTPRoute Path & Header Routing Engine
│
┌─────────┴─────────┐
▼ ▼
/v1/payments/* /v1/settlement/*
[ payment-core:8080 ] [ settlement-service:9090 ]

### Criteria and weights

| Criterion | Why it matters here | Weight | Source of the weight |
|---|---|---|---|
| Role-Oriented Separation & Portability | Platform SREs own TLS/port bindings (`Gateway`); feature squads own routing paths (`HTTPRoute`). | 0.35 | Marcus Vance (Lead Platform Lead) |
| True Client IP Preservation | PCI-DSS fraud detection requires accurate origin client IP attribution rather than proxy NAT IPs. | 0.30 | David O'Reilly (CISO SecOps) |
| Edge DDoS & Abuse Mitigation | Unthrottled traffic spikes would exhaust backend connection pools and database threads. | 0.20 | Enterprise Availability Mandate |
| Automated Cryptographic TLS Lifecycle | Manual certificate renewal processes inevitably result in production outage-inducing expirations. | 0.15 | Information Security Standard |


### Comparison

| Ingress Architecture Candidate | Routing API Standard | TLS Termination Seam | Client IP Attribution | Rate Limiting Engine | Evaluation |
|---|---|---|---|---|---|
| Option A: Legacy Ingress NGINX | Ingress v1 + Annotations | Ingress Controller | `X-Forwarded-For` string | Custom Lua annotations | Rejected: Annotation chaos; lacks typed Gateway API schema. |
| Option B: AWS ALB Ingress Controller | AWS Ingress Controller | AWS ALB (External) | ALB Header Injection | AWS WAF rate limits only | Rejected: Vendor lock-in; slow reconciliation during deployments. |
| Option C: Kubernetes Gateway API + Envoy (Chosen) | Gateway API (`HTTPRoute`) | In-Cluster Envoy Gateway | NLB PROXY Protocol v2 | Envoy Native Token Bucket | Selected: Open standard, role-isolated, low latency, portable. |


### Result

Option C is selected. Kubernetes Gateway API cleanly separates infrastructure routing ownership from service routing, fronted by high-performance Envoy Gateway instances.

---

### Required Mechanisms

#### 1. Gateway API Infrastructure & Listener Contract [MC-GW-01]
- **Gateway Resource**:
  ```yaml
  apiVersion: gateway.networking.k8s.io/v1
  kind: Gateway
  metadata:
    name: payment-external-gateway

namespace: ingress-system

spec:
gatewayClassName: eg
listeners:

      - name: https-payments

    protocol: HTTPS
    port: 443
    hostname: "api.payment.bank.internal"
    tls:
      mode: Terminate
      certificateRefs:

            - kind: Secret
              name: payment-api-tls-cert
        allowedRoutes:
          namespaces:
            from: Selector
            selector:
              matchLabels:
                ingress-routed: "true"

• GatewayClass: Backed by Envoy Gateway v1.0 controller.

2. HTTPRoute Specification & Path Routing [MC-HR-01]

• Product squads bind routes to the shared gateway via HTTPRoute:

  apiVersion: gateway.networking.k8s.io/v1
  kind: HTTPRoute
  metadata:
    name: payment-core-route
  

    namespace: payments-prod
  spec:
    parentRefs:

  - name: payment-external-gateway

        namespace: ingress-system
    hostnames:

  - "api.payment.bank.internal"
rules:
  - matches:
      - path:
          type: PathPrefix
          value: /v1/payments
    backendRefs:
      - name: payment-core-service
        port: 8080
        weight: 100
    timeouts:
      request: 5000ms
      backendRequest: 3500ms

#### 3. Client IP Preservation via PROXY Protocol v2 [MC-IP-01]
- **AWS NLB Annotation**: `service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"`
- **Envoy Configuration**: Decodes incoming PROXY Protocol v2 binary headers before HTTP parsing.
- Real client IP is populated into trusted `X-Forwarded-For` and `X-Real-IP` headers, preventing spoofing.

#### 4. Edge Token-Bucket Rate Limiting [MC-RL-01]
- Global rate limit filter enforced at Envoy Gateway:
- Base limit: 100 requests/second per source IP.
- Burst ceiling: 200 requests.
- Exceeded response: HTTP `429 Too Many Requests` with header `Retry-After: 1`.

#### 5. TLS Certificate Rotation Contract [MC-CR-01]
- cert-manager evaluates Let's Encrypt / DigiCert ACME certificates every 12 hours.
- Automated certificate renewal initiates when certificate validity < 30 days remaining.
- Envoy Gateway reloads modified TLS Secrets dynamically with zero connection drops.

---

### Invariants and Contracts

  Mandatory Gateway API Compliance [INV-ING-01]
    Production traffic routing must use Kubernetes Gateway API (`Gateway` and `HTTPRoute` CRDs).
    Deploying legacy `Ingress` v1 resources with bespoke controller annotations is prohibited.

  Strict PROXY Protocol Enforcement [INV-ING-02]
    Ingress controllers behind AWS NLBs must decode PROXY Protocol v2 headers. Direct unverified
    client IP assumption from TCP socket source addresses is strictly forbidden.

  Edge TLS Termination Floor [INV-ING-03]
    External HTTPS listeners must enforce TLS 1.3 as the default and TLS 1.2 as the minimum floor.
    Insecure cipher suites or unencrypted HTTP listeners (port 80) without automatic redirect are rejected.

## Explicit Unknowns

- Envoy Gateway memory utilization footprint under 14,000 active concurrent HTTP/2 long-lived streaming connections (G-1).
- DNS propagation latency during multi-region Route 53 NLB failover drills (G-2).

## Traceability

| Claim | Classification | Source | Freshness |
|---|---|---|---|
| Peak 14,000 HTTPS req/sec | provided | Traffic intake | Current |
| 4 AWS availability zones on EKS | provided | Infrastructure intake | Current |
| 15 backend payment microservices | provided | Scope intake | Current |
| Rejection of legacy Ingress v1 annotations | decided | Marcus Vance & David O'Reilly | 2026-09-15 |
| Kubernetes Gateway API adoption | decided | Architectural invariant INV-ING-01 | 2026-09-15 |
| PROXY Protocol v2 client IP preservation | decided | Architectural invariant INV-ING-02 | 2026-09-15 |


## Verification

No validator was supplied, so no command was run.

Reviewer self-check against ingress architecture standards:
- **API Modernity**: PASS. Uses Gateway API (`Gateway` and `HTTPRoute`) decoupling SRE and squad concerns.
- **Client IP Fidelity**: PASS. NLB PROXY Protocol v2 integration ensures untampered client IP attribution.
- **DDoS Resilience**: PASS. Token-bucket rate limiting (100 req/s burst 200) prevents backend starvation.
- **Markdown Hygiene**: PASS. Complies strictly with native Markdown rules in `rule_markdown.md`.

## Open Decisions

- `DEC-ING-01`: David O'Reilly to determine whether AWS WAF should inspect traffic directly at the AWS NLB or if Coraza WAF should run embedded inside Envoy Gateway (Owner: David O'Reilly).

## Next steps

1. Marcus Vance installs Envoy Gateway operator CRDs in the `ingress-system` namespace.
2. SecOps configures cert-manager ClusterIssuer for automated Let's Encrypt production certificates.
3. Feature squads migrate legacy Ingress YAML manifests to declarative `HTTPRoute` resources in staging.