rag cross tenant leakage test pack

The problem

Multi-tenant RAG systems often leak data through shared vector caches, improper session handling, or flawed retrieval logic. Standard unit tests miss these runtime isolation failures occurring between the embedding and generation phases.

What it does

• Evaluates RAG outputs for cross-tenant canary leakage using synthetic document sets.
• Scores retrieval accuracy across cold cache, warm cache, and active memory phases.
• Identifies failures where Tenant A queries return citations or context from Tenant B.
• Verifies positive controls to ensure retrieval failures are due to isolation, not system downtime.
• Exports structured evidence reports suitable for security audits and compliance reviews.

Frameworks & tools

Compatible with RAG architectures using vector databases, LangChain, or custom retrieval adapters. Includes Python-based scoring scripts and JSON input contracts.

Why this beats prompting it yourself

General LLMs tend to hallucinate security, often assuming isolation exists when it doesn't. This skill enforces a strict evidence discipline that treats missing data as a failure rather than a pass, ensuring you don't get a false sense of security from "clean" logs.

Use cases

• Pre-production security validation for multi-tenant SaaS applications.
• Regression testing after updating vector database indexing or RLS policies.
• Auditing LLM memory persistence during rapid session switching.
• Generating compliance-ready evidence for data isolation requirements.

Known limitations

This is not a replacement for database-level Row Level Security (RLS) reviews. Requires synthetic test tenants and canaries; do not use with production data or customer secrets.