Vector Database and Semantic Search Architect

Wrote docs/architecture/vector-search-architect/00-overview/vector-search-architect-overview.md and docs/architecture/vector-search-architect/verification/fitness-self-check.md. Complete vector search subsystem architecture specifying item-to-vector identity mapping, L2-normalized embedding-space pinning, mandatory pre-index tenant ACL filtering, hybrid BM25 + dense Reciprocal Rank Fusion, and atomic side-by-side index migration.

---

skill: vector-search-architect

Vector Search Architecture: B2B Marketplace Search Subsystem [VEC-B2B-001]

Summary

This specification establishes the vector similarity-search and hybrid indexing subsystem for the B2B Marketplace Search Subsystem under run ID b2b-vector-001. It indexes 28 million SKU items across 120 merchant tenants, sustaining peak query arrival of 2,400 queries/sec with an end-to-end p95 search latency budget <= 45 ms and candidate Recall@10 target >= 95.0%.

The architecture decisively resolves the catastrophic tenant isolation failure from incident VEC-4219 (where post-filtering 100 approximate nearest neighbor candidates by merchant_id in application memory collapsed recall to 18% for small-catalog tenants and leaked competitor catalog size through truncated result counts). It also prevents silent relevance collapse demonstrated in incident VEC-4220 (where in-place updates across differing embedding spaces mixed incompatible representations within a single index segment).

The subsystem enforces

1. Strict item-vector identity at (sku_id, locale) granularity;
2. Pinned embedding-space tuple space_id: text-embed-3-large@v1:1536:l2-norm;
3. Mandatory tenant and permission pre-filtering inside the HNSW index traversal graph;
4. Hybrid retrieval combining dense HNSW and sparse BM25 candidates via Reciprocal Rank Fusion (RRF, k=60);
5. Immutable dual-index side-by-side re-embedding with atomic pointer cutover.

Detailed Description

A vector search subsystem cannot be treated as a generic key-value datastore or an unconstrained KNN index. Storing and searching 28 million 1,536-dimensional vectors across 120 multi-tenant merchants introduces three critical failure modes:

1. Post-filtering under Approximate Nearest Neighbor (ANN) graphs produces empty or near-empty result sets when tenant density is skewed;
2. Unversioned embedding spaces silently corrupt similarity scores when query models or document encoders drift;
3. In-place index updates leave active search requests scoring vectors across disparate vector geometries during prolonged backfills.

Incoming Buyer Search Query ("heavy-duty industrial hydraulic pump")
                      │
     ┌────────────────┴────────────────┐
     ▼ (Dense Query Vectorization)     ▼ (Sparse Lexical Tokenization)
[ Text-Embedding-3-Large ]       [ BM25 Lexical Analyzer ]
  └── space_id: text-embed-3-      └── Tokens: {"hydraulic", "pump", "heavy"}
      large@v1:1536:l2-norm            └── Strict tenant_id predicate
     │                                 │
     ▼ (Pre-filtered ANN Traversal)    ▼ (Inverted Index Match)
[ HNSW Vector Index ]            [ Sparse Lexical Index ]
  ├── Payload Filter:              └── Filter: tenant_id == :merchant_id
  │     tenant_id == :merchant_id  └── Top-100 Lexical Candidates
  └── Top-100 Dense Candidates         │
     │                                 │
     └────────────────┬────────────────┘
                      ▼
[ Reciprocal Rank Fusion (RRF: k=60) & Tie-Breaker ]
  ├── Score: 1/(60 + Rank_dense) + 1/(60 + Rank_sparse)
  └── Candidate top-20 passed to downstream catalog rendering

Mechanism Specifications

1. Model or Agent Boundary:

   • Owner: AI Infrastructure Engineering (Marcus Vance).
   • Trigger: Ingestion of authorized SKU catalog item or incoming search query.
   • State/Algorithm: Embeddings must be generated using text-embed-3-large@v1 with fixed 1,536 dimensions and unit L2-normalization (||v||_2 = 1.0). Vectors without valid space_id header matching active space metadata are rejected at ingestion admission.
   • Failure Behavior: Ingestion jobs encountering missing or unparseable embedding payloads discard the chunk to dead-letter queue vec.sku.dlq without mutating index segments.
   • Test Oracle: Automated dimension and normalization assertion verifying dim(v) == 1536 and |dot(v, v) - 1.0| < 1e-5.

2. Tool Policy and Tenant Filtering:

   • Owner: Search Platform Core / SecOps.
   • Trigger: Search request execution.
   • State/Algorithm: Mandatory tenant isolation is enforced at index traversal time (pre-filter). During HNSW beam traversal, candidate nodes lacking matching tenant_id attribute are skipped before distance scoring. Post-filtering retrieved top-k results is strictly forbidden.
   • Failure Behavior: Query requests lacking authenticated tenant_id or passing wildcard merchant scopes abort immediately with error ERR_TENANT_FILTER_MANDATORY.
   • Test Oracle: Penetration probe confirming zero candidates from foreign tenant_id appear in retrieved candidate sets under any query distribution.

3. Evaluation Oracle:

   • Owner: Head of Search Relevance (Elena Rostova).
   • Trigger: Pre-deployment CI pipeline and nightly drift audit.
   • State/Algorithm: Factual evaluation against curated golden benchmark of 1,200 domain-specific B2B purchasing queries with human-labeled relevance judgments. Self-graded LLM evaluation is rejected; evaluation compares ANN retrieval candidate sets against exact brute-force flat L2-normalized cosine scan baselines.
   • Failure Behavior: Candidate sets exhibiting Recall@10 < 95.0% or NDCG@10 < 0.88 block index artifact promotion.
   • Test Oracle: Executable Python verification suite eval_retrieval_metrics.py asserting Recall@10 >= 0.950 against exact baseline ground truth.

4. Context Budget and Candidate Volume:

   • Owner: Search Platform Operations.
   • Trigger: Upstream query dispatch from API Gateway.
   • State/Algorithm: Individual search queries retrieve at most 100 dense and 100 sparse candidates. Reciprocal Rank Fusion (RRF) deduplicates and emits exactly top-20 candidate SKUs to downstream presentation tiers.
   • Failure Behavior: Upstream requests attempting top_k > 100 are truncated to 100 at the API boundary with warning telemetry.
   • Test Oracle: Gateway test confirming retrieval response payload memory allocation <= 64 KiB per request.

Alternatives rejected

Contracts and Invariants

Item Identity Granularity [INV-VEC-01]
  Each vector record maps to exactly one canonical (sku_id, locale). Vectors must not be generated
  per product variant (e.g., color or packaging size) to prevent identical parent copy from
  monopolizing top-k result slots.

Embedding Space Invariance [INV-VEC-02]
  Every vector record and search query must declare space_id: "text-embed-3-large@v1:1536:l2-norm".
  Comparing a query vector from one space against vectors indexed under another space is blocked
  at the query broker layer with status ERR_INCOMPATIBLE_EMBEDDING_SPACE.

Mandatory Pre-Index Tenant Filtering [INV-VEC-03]
  Tenant isolation must be evaluated as an active filter predicate during HNSW index graph traversal.
  Post-filtering retrieved ANN candidates in application memory is prohibited.

Zero Downtime Side-by-Side Re-Embedding [INV-VEC-04]
  Embedding model migrations or space updates must construct a secondary shadow index alongside
  the active index. Search traffic routes to the active index until the shadow index passes full
  evaluation gates and is promoted atomically via cluster collection alias flip.

Recall Safety Floor [INV-VEC-05]
  Approximate nearest neighbor retrieval must maintain Recall@10 >= 95.0% relative to exact
  unquantized brute-force cosine baselines across all 120 tenant catalog distributions.

Ownership and Handoffs

Traceability

Verification

No validator was supplied, so no command was run.

Reviewer self-checks against vector search architecture standards:

Baseline Rigor: PASS. Non-vector baselines (exact match, lexical BM25) evaluated; hybrid search explicitly reserved for verified lexical gap.

Identity & Space Pinning: PASS. Item identity anchored to (sku_id, locale); embedding space pinned with model, version, dimension (1536), and L2-norm.

Filtering Safety: PASS. Tenant boundaries pre-filtered inside HNSW graph traversal; zero post-filtering candidate leakage.

Migration Determinism: PASS. Re-embedding mandates isolated side-by-side index builds with atomic pointer cutover.

• Markdown Hygiene: PASS. Native Markdown syntax strictly adheres to rule_markdown.md.

Open Decisions

• DEC-VEC-01: Elena Rostova to determine whether cross-encoder re-ranking (e.g. BGE-Reranker-Large) should be applied to the top 20 candidates for high-margin category searches within a 15 ms latency budget (Owner: Elena Rostova).

---

skill: vector-search-architect

B2B Marketplace Vector Search — Fitness Self-Check [VEC-FIT-001]

Summary

This fitness self-check evaluates the vector search subsystem architecture for the B2B Marketplace Search Subsystem against the three red-capable domain failure probes: unbounded autonomy, self-graded evaluation, and prompt-only control. All targeted probes pass by design construction. A self-check is supporting evidence, never the authoritative gate. Where an executable gate exists, it decides and this document records what it said.

Detailed Description

Residual Risk

• High-density tenant index skew: if a single merchant accounts for > 40% of all 28M items, HNSW entry-point clustering may cause a 6 ms latency penalty during highly selective pre-filtered traversals. Accepted by Marcus Vance pending partition-sharding evaluation in Q4.

Traceability

Verification

No validator was supplied, so no command was run.

Open Decisions

None.

Next steps

1. Marcus Vance provisions 8-node memory-optimized vector database cluster with HNSW index configuration.
2. Search Platform Core implements pre-filtering query broker plugin enforcing typed tenant_id parameter binding.
3. Elena Rostova executes baseline evaluation suite eval_retrieval_metrics.py against the 1,200 golden query benchmark.