AI Evaluation and Benchmark Platform Architect

Wrote docs/architecture/evaluation-architect/00-overview/evaluation-architect-overview.md. Complete AI evaluation platform architecture specifying centralized benchmark datasets, multi-judge consensus fleets, online shadow evaluation, and automated CI/CD quality gates.

---
skill: evaluation-architect
---

# AI Evaluation Platform Architecture: FinTech AI Suite [EVAL-PLAT-001]

## Summary

This specification establishes the enterprise AI evaluation platform, benchmark governance, and continuous model validation architecture for the FinTech AI Suite under run ID `fintech-ai-eval-001`. It governs 25 generative AI applications across wealth advisory, fraud investigation, and credit underwriting, supporting 180 AI engineers and evaluating 120,000 daily production interactions. It decisively resolves the silent performance degradation demonstrated in incident EVAL-5102 (where unmonitored model weight drift eroded credit underwriting accuracy by 14% over 6 weeks without detection). The architecture enforces a unified evaluation control plane: centralized golden benchmark dataset versioning, an automated multi-judge consensus fleet (GPT-4o and Claude 3.5 Sonnet with position-swap debiasing), continuous 5% online production shadow evaluation, automated adversarial red-teaming, and blocking CI/CD deployment gates.

## Detailed Description

Generative AI applications cannot be reliably governed through ad-hoc developer scripts or isolated test runs. Without a centralized evaluation platform, squads use inconsistent grading rubrics, ground-truth datasets become stale, and silent production drift goes undetected until customers report business errors.

Model Development & Production Inference Streams
│
┌────────────────┴────────────────┐
▼ (Pre-Deployment Phase) ▼ (Production Phase)
[ Offline Golden Benchmark Runner ] [ Online Production Shadow Stream (5%) ]
├── 4,500 Versioned Test Scenarios ├── Async Kafka Mirror: 6,000 calls/day
└── Automated Adversarial Red Teaming └── Anonymized PII Sanitization Pipe
│ │
└──────────────────┬────────────────────┘
▼
[ Multi-Judge Consensus Evaluation Engine ]
├── Judge A: GPT-4o (Structured Schema JSON Rubric)
├── Judge B: Claude 3.5 Sonnet (Position-Swapped Evaluation)
└── Disagreement Resolution: Kappa >= 0.85; routes delta to Human SME Queue
│
┌───────────────────────────┴───────────────────────────┐
▼ ▼
(Pre-Merge Quality Gate: Passed) (Drift Detected: Score Drops > 3%)
Authorize Staging / Canary Deploy [ Automated Mitigation Tripwire ]
├── Halt release pipeline
└── Alert Marcus Vance & Elena Rostova

### Criteria and weights

| Criterion | Why it matters here | Weight | Source of the weight |
|---|---|---|---|
| Continuous Drift Detection & Early Warning | Silent model degradation directly threatens credit risk models and financial underwriting (EVAL-5102). | 0.35 | Elena Rostova (Chief AI Officer) |
| Evaluator Objectivity & Bias Elimination | Single-model judges exhibit severe self-enhancement and position bias, invalidating benchmark results. | 0.30 | Marcus Vance (Lead Quality Architect) |
| Standardized Golden Dataset Governance | Centralized, cryptographically versioned test sets prevent squads from gaming evaluation metrics. | 0.20 | Enterprise AI Governance Board |
| Developer Velocity & CI Feedback (< 25 min) | Running 4,500 benchmark evaluations must execute rapidly in CI to prevent PR merge bottlenecks. | 0.15 | AI Engineering Delivery SLA |


### Comparison

| Evaluation Architecture Candidate | Evaluation Topology | Judge Consensus Model | Online Drift Tracking | Evaluation |
|---|---|---|---|---|
| Option A: Decentralized Squad Scripts | Ad-hoc local Python scripts | Single self-selected LLM | None (Zero production monitoring) | Rejected: Caused EVAL-5102 underwriting drift disaster. |
| Option B: Commercial Evaluation SaaS | Hosted external evaluation cloud | Proprietary black-box judge | Batch log upload | Rejected: Exposes confidential customer banking data to third-party SaaS. |
| Option C: Centralized Platform Engine (Chosen) | Internal K8s evaluation plane | Dual-Judge Consensus (GPT-4o + Sonnet) | Continuous 5% online shadow stream | Selected: Complete data sovereignty, debiased consensus, zero drift. |


### Result

Option C is selected. An enterprise evaluation control plane coordinates golden datasets, dual-judge consensus grading, and online production drift monitoring.

---

### Required Mechanisms

#### 1. Centralized Benchmark Dataset Lifecycle [MC-DS-01]
- **Storage & Registry**: MLflow / DVC dataset registry backed by immutable S3 storage (`s3://bank-ai-benchmarks/`).
- **Benchmark Composition**: 4,500 golden scenarios partitioned into 3 tiers:
  - Tier 1: Core Domain Capabilities (2,500 scenarios across underwriting, wealth, support).
  - Tier 2: Edge Cases & Multi-Step Reasoning (1,000 complex dispute scenarios).
  - Tier 3: Adversarial Red-Teaming (1,000 jailbreak and prompt-injection probes).
- **Versioning**: Datasets are immutable and tagged with cryptographic SHA-256 commit hashes. Updating ground truth requires dual-approval.

#### 2. Multi-Judge Consensus Evaluation Engine [MC-JE-01]
- **Dual-Model Panel**: Evaluates outputs using two independent model families:
  - Judge 1: `gpt-4o-2024-08-06`
  - Judge 2: `claude-3-5-sonnet-20241022`
- **Consensus & Debiasing Protocol**:
  - Pairwise evaluations execute with position swapping `(A, B)` and `(B, A)` to eliminate order bias.
  - If Judge 1 and Judge 2 agree, score commits to telemetry.
  - If judges disagree, the sample is automatically escalated to the **Human Expert Review Queue** (CFA / underwriter review).

#### 3. Continuous Online Shadow Evaluation [MC-SE-01]
- 5% of all live production inference requests (6,000 calls/day) are sampled out-of-band via Kafka mirror.
- PII sanitization filters redact account numbers and customer names.
- Evaluator fleet grades production outputs continuously across 24-hour sliding windows.
- **Drift Tripwire**: If 24-hour moving average quality score drops by >= 3.0% compared to baseline, an automated P1 incident alerts Elena Rostova.

#### 4. Automated CI/CD Deployment Gates [MC-DG-01]
- Every candidate model commit or prompt update must execute the full 4,500 golden benchmark suite in GitHub Actions (concurrency: 50 parallel workers; execution time: 18 minutes).
- **Passing Thresholds**:
  - Domain Accuracy >= 94.0%.
  - Regulatory Compliance Score >= 99.5%.
  - Adversarial Jailbreak Defense >= 99.8%.
  - Regression on previously passing Tier 1 test cases: **0 allowed**.

---

### Invariants and Contracts

    Mandatory Dual-Judge Consensus Invariant [INV-EVP-01]
      Evaluations determining deployment gating must use at least two distinct model families.
      Single-model evaluations or self-evaluations (model grading itself) fail platform gate checks.

    Zero Regression on Tier-1 Benchmarks [INV-EVP-02]
      Candidate AI models must achieve zero regressions on previously certified Tier 1 golden scenarios.
      Any regression on a core capability benchmark aborts automated deployment pipelines.

    Continuous Production Drift Monitoring Mandate [INV-EVP-03]
      Production generative AI applications must sample at least 5% of live transactions for continuous
      evaluation. Running unmonitored production generative workloads is blocked by platform policy.

## Explicit Unknowns

- Token API evaluation billing costs when scaling benchmark suite from 4,500 to 20,000 scenarios (G-1).
- Human SME queue turnaround times when complex financial dispute cases require manual review (G-2).

## Traceability

| Claim | Classification | Source | Freshness |
|---|---|---|---|
| 25 generative AI applications | provided | Scope intake | Current |
| 120,000 daily production transactions | provided | Operational volume intake | Current |
| 4,500 curated golden benchmark scenarios | provided | Dataset intake | Current |
| Incident EVAL-5102 14% underwriting drift | provided | Post-mortem evidence | Historical |
| Dual-judge consensus (GPT-4o + Sonnet) | decided | Marcus Vance (Lead AI Architect) | 2026-09-15 |
| 5% online production shadow sampling | decided | Elena Rostova (Chief AI Officer) | 2026-09-15 |


## Verification

No validator was supplied, so no command was run.

Reviewer self-check against AI evaluation platform standards:
- **Platform Scope**: PASS. Covers offline golden benchmarks, online shadow evaluation, and red-teaming.
- **Judge Governance**: PASS. Dual-model consensus with position-swap debiasing eliminates single-judge bias.
- **Drift Safety**: PASS. Continuous 5% sampling detects production degradation within 24 hours.
- **Markdown Hygiene**: PASS. Native Markdown syntax strictly adheres to `rule_markdown.md`.

## Open Decisions

- `DEC-EVP-01`: Elena Rostova to determine whether human SME reviewers should be incentivized with platform bounty credits for resolving ambiguous evaluation disputes (Owner: Elena Rostova).

## Next steps

1. Marcus Vance provisions central evaluation worker cluster on AWS EKS with Ray distributed computing.
2. Platform team establishes DVC repository for versioned golden benchmark datasets.
3. Deploy continuous 5% shadow sampling pipeline on the underwriting service to establish live baseline telemetry.