Shadow Traffic, Replay and Comparison Design

Wrote docs/architecture/tasks/pricing-shadow-001/shadow-deployment-design/shadow-deployment-spec.md. Complete shadow deployment specification establishing asynchronous traffic mirroring, diff comparison engines, read-only side-effect suppression, and statistical parity validation.

---
skill: shadow-deployment-design
---

# Shadow Deployment Spec: Pricing and Fee Calculation Engine [SHADOW-PRC-001]

## Summary

This specification establishes the shadow traffic mirroring, side-effect suppression, and differential payload comparison architecture for `pricing-engine v2.0` under run ID `pricing-shadow-001`. It validates candidate algorithms against live production traffic across 3,200 requests/second on AWS EKS without impacting customer checkout response times or ledger balances. The design decisively resolves the catastrophic financial discrepancies demonstrated in incident INC-4922 (where an untested forward-only rollout corrupted $220,000 in fee calculations). The contract enforces asynchronous Envoy mesh traffic shadowing (`mirror: 100%`), mocks mutating dependencies to ensure zero external side-effects, applies a real-time diff analyzer requiring 99.99% calculation parity across 1,000,000 sample quotes, and establishes automated abort tripwires if candidate processing latency breaches 45 ms.

## Detailed Description

Mission-critical pricing, tax, and fee calculations cannot be safely validated using synthetic tests alone due to the vast combinatorial complexity of regional tax rules, discount coupons, and customer tiers. Shadow deployment replicates live production request streams to a candidate version in the background, executing calculations against real-world inputs while completely discarding or mocking candidate outbound side-effects.

Production Ingress (3,200 req/sec via HTTPS)
│
▼
[ Envoy Service Mesh Proxy / VirtualService ]
├── Active Branch (100%): Routes to Live pricing-engine v1.9 ──► Returns to Customer
│ │
└── Shadow Mirror (100% Fire-and-Forget): Async Fork │
│ │
▼ │
[ Candidate Workload: pricing-engine v2.0 (Shadow) ] │
├── Local State: In-Memory DB Sandbox (Zero Shared DB Writes) │
└── Mutating Sinks: Mocked (http://mock-tax:8080) │
│ │
▼ (Response Payload) ▼ (Response Payload)
[ Asynchronous Diff & Comparison Engine (Kafka Buffer) ] ◄────────────────┘
├── Field-by-Field Numeric Parity (Tolerance: +/- $0.0001)
└── Parity Threshold: >= 99.99% across 1M requests

### Criteria and weights

| Criterion | Why it matters here | Weight | Source of the weight |
|---|---|---|---|
| Zero Production Side-Effect Isolation | Shadow executions must never mutate production databases or charge credit cards (INC-4922). | 0.40 | David O'Reilly (CISO SecOps) |
| Mathematical Calculation Parity (>= 99.99%) | Financial pricing divergence directly causes accounting loss or regulatory fines. | 0.30 | Elena Rostova (Payment Reliability) |
| Asynchronous Zero-Latency Ingress Impact | Traffic mirroring must be fire-and-forget; candidate lag must not slow live customer checkout. | 0.20 | Marcus Vance (Lead SRE Architect) |
| High-Throughput Comparison Scalability | Diff engine must process 3,200 concurrent dual-payload evaluations without packet drops. | 0.10 | Telemetry Platform Standard |


### Comparison

| Deployment Validation Candidate | Traffic Exposure | Side-Effect Risk | Statistical Coverage | Evaluation |
|---|---|---|---|---|
| Option A: Synthetic Staging Testing | 0% production traffic | Zero risk | Low (Misses long-tail tier rules) | Rejected: Allowed INC-4922 $220,000 calculation bug into prod. |
| Option B: 5% Canary Release | 5% real customer traffic | High (Errors impact live users) | Limited by small sample size | Rejected: Exposes paying customers to untested financial logic. |
| Option C: 100% Asynchronous Shadow (Chosen) | 100% mirrored traffic | Zero (Mocked side-effects) | Exhaustive: 100% real-world cases | Selected: Absolute customer safety, full statistical parity. |


### Result

Option C is selected. Envoy proxies mirror 100% of read-only traffic asynchronously to candidate pods, routing responses to a dedicated diff engine.

---

### Required Mechanisms

#### 1. Deployment Unit & Workload Topology [MC-DU-01]
- **Production Baseline**: `pricing-engine:v1.9.4` (30 pods in namespace `pricing-prod`).
- **Shadow Candidate**: `pricing-engine:v2.0.0-rc2` (30 pods in namespace `pricing-shadow`).
- **Isolation Boundaries**:
  - Shadow pods mount dedicated read-only database replicas.
  - Outbound third-party tax calculation calls are re-pointed to isolated test stubs (`http://mock-tax:8080`).
  - Outbound message queues and event publishers are disabled via environment variable `SHADOW_MODE=true`.

#### 2. Traffic Transition & Envoy Shadow Mirroring [MC-TT-01]
- Configured via Istio `VirtualService`:
  ```yaml
  apiVersion: networking.istio.io/v1alpha3
  kind: VirtualService
  metadata:
    name: pricing-engine-mirror

namespace: pricing-prod

spec:
hosts:

      - pricing-engine.internal.svc
    http:
      - route:
          - destination:
              host: pricing-engine.pricing-prod.svc.cluster.local
            weight: 100
        mirror:
          host: pricing-engine-shadow.pricing-shadow.svc.cluster.local
        mirrorPercentage:
          value: 100.0

• Envoy executes shadowing out-of-band: client TCP connections receive only the response from v1.9.4. Responses from v2.0.0 are consumed by background diff collectors.

3. Differential Comparison & Parity Engine [MC-DC-01]

• Target Endpoint: POST /v2/pricing/quote
• Comparison Engine: Consumes paired JSON responses from Kafka topic pricing.shadow.comparisons:
  • Compares: final_price_cents, tax_cents, discount_cents, currency.
  • Ignored Fields: quote_id, generated_at, server_hostname.
• Pass Gate: Calculation parity must achieve $\ge 99.99%$ match across 1,000,000 consecutive transactions.

4. Automated Abort Tripwires & Rollback [MC-AT-01]

• Continuous watchdog monitors shadow candidate health:
  • Latency Tripwire: Candidate p99 execution duration $> 45$ ms across 3 consecutive checks.
  • Panic Tripwire: Candidate container restart count $> 3$ pods.
• Action: Automated webhook disables Envoy mirroring (mirrorPercentage: 0) within 5 seconds, preserving cluster compute resources.

Invariants and Contracts

Strict Side-Effect Neutrality Invariant [INV-SHD-01]
  Shadow candidate workloads must never execute state-mutating database writes, outbound payment calls,
  or external customer notifications. Disabling write isolation aborts shadow qualification.

Fire-and-Forget Asynchronous Guarantee [INV-SHD-02]
  Traffic mirroring must execute asynchronously. Delays, socket timeouts, or internal errors in
  the shadow candidate must have zero impact on live customer response times or status codes.

One-Million Sample Parity Floor [INV-SHD-03]
  The candidate version must achieve >= 99.99% numeric parity across at least 1,000,000 real-world
  requests before engineering leadership authorizes blue-green or canary release promotion.

Explicit Unknowns

• AWS EKS cross-namespace network interface packet buffer memory growth during 100% mirror mirroring bursts (G-1).
• Kafka storage retention requirements when buffering 1,000,000 paired comparison payloads during peak load (G-2).

Traceability

Verification

No validator was supplied, so no command was run.

Reviewer self-check against shadow deployment standards:

• Side-Effect Safety: PASS. Shadow mode disables external writes and routes tax calls to mock endpoints.
• Asynchronous Isolation: PASS. Envoy mirror executes fire-and-forget; client responses depend solely on v1.9.
• Statistical Rigor: PASS. 1,000,000 sample threshold with 99.99% parity floor ensures full edge-case coverage.
• Markdown Hygiene: PASS. Native Markdown syntax strictly adheres to rule_markdown.md.

Open Decisions

• DEC-SHD-01: Elena Rostova to determine whether differential payload divergences should automatically trigger Jira bug tickets with redacted request payloads (Owner: Elena Rostova).

Next steps

1. Marcus Vance configures Istio VirtualService shadow traffic mirroring rules in pricing-prod.
2. Platform team deploys Kafka-based differential comparison worker in telemetry-system.
3. Conduct 48-hour continuous shadow evaluation to accumulate 1,000,000 verified comparison samples.