Diagnosing RAG Failure Modes

RAG pipelines often fail silently, producing plausible answers that lack factual grounding or miss critical context. This skill provides a structured diagnostic framework for engineers to investigate why a retrieval system is underperforming. It moves beyond simple similarity scores to analyze multi-hop logic, causal links, and context assembly errors.

What it does

• Evidence Mapping breaks down user queries into atomic facts and multi-hop requirements to identify what the retriever missed.
• Retrieval Tracing audits chunks, ranks, and metadata filters to find recall gaps or boundary failures.
• Grounding Audits maps generated claims to specific retrieved evidence, labeling them as supported, unsupported, or contradicted.
• Failure Classification categorizes issues into specific modes like query decomposition failure, ranking errors, or source conflicts.
• Remediation Planning generates testable fixes tied to specific mechanisms rather than general best practices.

How it works

1. Supply artifacts such as the user query, retrieved chunks, and the generated answer with citations.
2. Review the evidence map to see the logical hops and facts required to answer the query correctly.
3. Analyze the diagnostic report which identifies the primary failure mode and confidence levels.
4. Implement the regression tests provided to validate that proposed fixes resolve the specific retrieval or grounding gap.

Frameworks & tools

Works with any RAG stack including LangChain, LlamaIndex, Haystack, or custom Python implementations. Compatible with vector databases like Pinecone, Weaviate, or Milvus, and reranking models like Cohere or BGE.

Why this beats prompting it yourself

Standard prompts often hallucinate the cause of RAG failure or rely too heavily on high similarity scores. This skill enforces a rigorous evidence-to-claim mapping process and distinguishes between observed facts and hypotheses, ensuring you fix the root cause instead of symptoms.

Use cases

• Multi-hop reasoning where the agent fails to connect information across two different documents.
• Causal analysis when the pipeline identifies a correlation but lacks the evidence to prove a causal link.
• Citation validation to ensure the LLM is not "hallucinating" sources that don't actually support the claim.

Known limitations

Requires user-provided retrieval traces and documents to function. It cannot independently execute the pipeline or verify production changes.