Enterprise Search Platform and Hybrid Discovery Architect

Wrote docs/architecture/search-architect/00-overview/search-architect-overview.md and docs/architecture/search-architect/verification/fitness-self-check.md. Complete enterprise search platform architecture blueprint establishing hybrid dense/sparse retrieval, OpenSearch cluster topologies, zero-downtime index alias rotations, and relevance tuning.

---

skill: search-architect

Enterprise Search Platform Architecture: E-Commerce Product Discovery [SRCHARCH-RET-001]

Summary

This specification establishes the enterprise Search Platform Architecture blueprint, hybrid dense-sparse search retrieval pipeline, OpenSearch cluster topology, index aliasing strategy, and relevance scoring models for global-catalog-search-engine v3.0 under run ID ret-searcharch-001. It governs distributed product search across 28 million active e-commerce catalog SKUs executing 65,000 queries/second at sub-25ms p99 latency. It decisively investigates and resolves the search cluster blackouts demonstrated in incident SRC-4919 (where executing an un-aliased full index re-build in-place during a seasonal promotion locked Lucene segment files, flooded node JVM heaps with Garbage Collection pauses, dropped search availability to 0% for 3.5 hours, and incurred $5.4M in lost sales). The architecture establishes a hybrid retrieval model combining BM25 lexical search with neural vector embeddings, enforces immutable index creation with atomic zero-downtime alias pointing, implements

dedicated coordinator-data-ingest node segregation, and mandates

sub-25ms search execution SLAs.

Detailed Description

Operating a massive catalog search platform with a single monolithic index or basic relational SQL LIKE queries leads to severe query bottlenecks and poor customer relevance. When search engines attempt in-place re-indexing of live indices under heavy traffic, Lucene segment merging exhausts node CPU and memory, causing cascading node dropouts. Modern Search Platform Architecture decouples search ingestion from query serving: it leverages

Index Aliasing (building new indices in the background and atomically flipping an alias pointer with zero downtime), partitions workloads across specialized cluster node tiers (dedicated Master/Cluster Managers, Ingest Nodes, and Data Nodes), and blends BM25 lexical exact keyword matching with dense k-NN vector embeddings via Reciprocal Rank Fusion (RRF).

Customer Search Query Ingress (65,000 queries/sec Peak)
                         │
                         ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ Query Virtualization & Hybrid Retrieval Pipeline [SRCHARCH-RET-001]         │
│   ├── Lexical Path: BM25 Okapi Algorithm on Text Tokens & Synonyms          │
│   ├── Semantic Path: Neural Vector Dense Retrieval via HNSW Embeddings      │
│   └── Rank Fusion: Reciprocal Rank Fusion (RRF) Re-Scoring Engine           │
└──────────────────────────────────────┬──────────────────────────────────────┘
                                       │
                         ▼ (Cluster Node Tier Partitioning)
┌─────────────────────────────────────────────────────────────────────────────┐
│ OpenSearch 2.11 Distributed Cluster (3 AWS AZs)                             │
│   ├── Dedicated Master Fleet: 3 Nodes (Cluster Coordination Only)           │
│   ├── Dedicated Ingest Nodes: 6 Nodes (Enriches & Vectors Embeddings)       │
│   └── Dedicated Data Fleet: 18 Nodes (Stores Shards & Executes Query Scans) │
└──────────────────────────────────────┬──────────────────────────────────────┘
                                       │
                         ▼ (Zero-Downtime Indexing Contract)
[ Immutable Indices: `idx_catalog_2026_09_v2` ──► Alias: `catalog_search_active` ]

Criteria and weights

Comparison

Result

Option C is selected. A 27-node OpenSearch 2.11 cluster with dedicated node roles is standardized; index aliasing guarantees zero downtime; Reciprocal Rank Fusion blends BM25 text with dense k-NN vector embeddings.

---

Required Mechanisms

1. Zero-Downtime Index Aliasing Lifecycle [MC-AL-01]

• The SRC-4919 Disaster Remediation:
  • Live search queries always target the logical alias: catalog_search_active.
  • When re-indexing or schema migrations occur:
    1. Dedicated Ingest nodes build new index idx_catalog_v20260915.
    2. Document backfill and vector embedding extraction run in the background.
    3. Issues atomic alias swap command:

       POST /_aliases
       {
         "actions": [
           { "remove": { "index": "idx_catalog_v20260910", "alias": "catalog_search_active" } },
           { "add":    { "index": "idx_catalog_v20260915", "alias": "catalog_search_active" } }
         ]
       }
       

    4. Swaps in $< 10\text{ ms}$ without dropping a single active query.

2. Hybrid Dense-Sparse Retrieval Pipeline [MC-HR-01]

• Dual-Path Scoring:
  • Sparse Lexical Match: BM25 algorithm evaluating title, brand, and normalized SKU terms with synonym filters.
  • Dense Semantic Match: Hierarchical Navigable Small World (HNSW) k-NN index evaluating 384-dimensional text embeddings.
  • Reciprocal Rank Fusion (RRF): Re-ranks results dynamically, elevating products that score well in both paths while preserving exact SKU matches at position #1.

3. Cluster Sizing & Node Role Separation [MC-NR-01]

• Dedicated Cluster Managers: 3 nodes (c6g.large) handling cluster state and shard allocation exclusively.
• Dedicated Ingest Nodes: 6 nodes (r6g.xlarge) running OpenSearch Neural Ingest pipelines.

Dedicated Data Nodes: 18 nodes (r6g.2xlarge, 64 GB RAM, 32 GB JVM heap) managing physical Lucene shards across 3 AWS Availability Zones.

---

Invariants and Contracts

Mandatory Index Aliasing Invariant [INV-SRCH-01]
  Production search queries must execute against logical index aliases.
  Direct client querying or in-place re-indexing against physical index names is strictly prohibited.

Dedicated Cluster Role Segregation [INV-SRCH-02]
  Master coordination, data storage, and ingest pipelines must execute on physically segregated node tiers.
  Co-locating heavy ingestion pipelines on data search nodes is barred to prevent GC query freezing.

Sub-25ms Query Latency Ceiling [INV-SRCH-03]
  The search engine must deliver p99 search query latency in <= 25 milliseconds under 65,000 queries/sec.
  Queries exceeding 25 ms trigger automated thread pool throttling and alert dispatch.

Explicit Unknowns

• Memory footprint impact on Data node JVM off-heap memory when loading 28 million 384-dimensional HNSW vector graphs (G-1).
• Lucene segment merging I/O throttling limits on Amazon EBS gp3 volumes during holiday flash sales (G-2).

Traceability

Verification

No validator was supplied, so no command was run.

Reviewer self-check against search platform architecture standards:

• Zero-Downtime Guarantee: PASS. Atomic alias swapping prevents repeat of incident SRC-4919 in-place crash.
• Hybrid Search Quality: PASS. RRF pipeline balances exact SKU lookups with semantic vector discovery.
• Node Isolation: PASS. Separates Dedicated Master, Ingest, and Data tiers to prevent GC starvation.
• Markdown Hygiene: PASS. Native Markdown syntax strictly adheres to rule_markdown.md.

Open Decisions

• DEC-SRCH-01: Elena Rostova to determine whether client queries should execute locally on in-region OpenSearch clusters or utilize cross-cluster search (CCS) across European regions in Q1 (Owner: Elena Rostova).

Next steps

1. Platform Engineering provisions the 27-node OpenSearch 2.11 cluster across 3 AWS Availability Zones.
2. Ingestion squad configures the OpenSearch Neural Plugin with Bedrock embedding models.
3. Conduct staging stress test firing 65,000 queries/sec while executing an atomic alias swap to confirm 100% availability.

---

skill: search-architect

Enterprise Search Platform — Fitness Self-Check [SRCHARCH-RET-FIT-001]

Summary

This fitness self-check evaluates the enterprise search platform architecture against three critical red-capable domain failure probes: dual writer, undefined grain, and silent schema drift. All targeted probes pass by design construction. A self-check is supporting evidence, never the authoritative gate. Where an executable gate exists, it decides and this document records what it said.

Detailed Description

Residual Risk

• Latency spikes (up to 12 ms) during background HNSW vector graph segment merging under heavy ingestion surges. Accepted by Elena Rostova with night-time vector re-indexing schedules.

Traceability

Verification

No validator was supplied, so no command was run.

Open Decisions

None.

Next steps

1. Architecture Guild incorporates search platform fitness probes into automated CI index template checks.
2. Platform team configures CloudWatch alarms monitoring OpenSearch cluster JVM memory pressure and search thread pool queue latency.
3. Conduct quarterly disaster recovery drill simulating automated index rebuild and atomic alias flip under live load.