Diagnosing Rag Failure Modes

Problems It Solves Silent retrieval failure — RAG pipelines return plausible-sounding results on multi-hop and causal queries, making failures hard to detect.

$10

30-day refund guarantee

Secure checkout via Stripe

5 installsSecurity scanned

Works with the AI tools you already use

CCCGGVWM+14 more

See it in action

You say

Run a RAG failure diagnosis on this failing query: 'What led to the 2023 DB migration?' The agent currently returns the migration plan but misses the causal context and sequence.

Your agent does

QUERY: What led to the 2023 DB migration? CLASS: Class B Relational/Temporal FAILURE: Pattern 2 (Temporal) & Pattern 3 (Org Context) ROOT CAUSE: RAG retrieved migration docs but missed the Slack discussion nodes explaining 'why' and the sequence of 3 preceding outages. FIX: Timeline Index.

What you get

RAG pipeline debugging: An agent over internal documentation fails on "Why did we deprecate the v1 API?" — a query that requires linking the deprecation notice, the downstream services affected, and the architectural rationale from a decision record written two years earlier. The diagnostic checklist scores it as Class B (3 checks: multi-document join, causal chain, temporal span). Root cause: structural RAG mismatch. Fix: knowledge graph traversal.Architecture investment justification: A team wants to add a knowledge graph but needs to demonstrate to engineering leadership why the current vector store cannot be tuned to handle the failing queries. The failure classification report provides a structured argument with root cause analysis and specific pattern attribution.Onboarding agent quality review: A new onboarding assistant answers "What is our PTO policy?" correctly but fails on "Why is our engineering team structured the way it is?" The diagnostic separates these as Class A and Class B respectively — and identifies that the second query requires organizational context provenance that was never ingested, not better embeddings.Vendor evaluation: A team is evaluating RAG vendors and receives demo results on their sample queries. Running the diagnostic checklist against the sample set reveals that all demo queries were Class A. Their actual production queries are 60% Class B. The vendor's system is being benchmarked on a task distribution it will never face in production.

About this skill

Problems It Solves

Silent retrieval failure — RAG pipelines return plausible-sounding results on multi-hop and causal queries, making failures hard to detect. Teams iterate on embedding quality and chunking strategy for weeks before realizing the query type is the problem, not the implementation.
Wrong fix applied — Most RAG debugging focuses on embedding models, chunk size, and reranking. These are the right levers for factual lookup failures. They do nothing for relational and temporal failures, where the architecture itself is mismatched to the query.
Query type blindness — No standard vocabulary exists for distinguishing "what is X" from "how did X come to be" at the pipeline level. Without this distinction, every query gets routed to the same retrieval system regardless of structural fit.
Scale degradation — RAG degrades on large corpora not because the embeddings get worse, but because the signal-to-noise ratio collapses. Teams add reranking layers and see marginal improvement, missing that tiered retrieval is the actual fix.

What You Get

The two-class query taxonomy — A clear, actionable split between Class A (factual lookup, RAG-safe) and Class B (relational/temporal, RAG danger zone), with concrete examples of each so classification is fast and unambiguous.
Five-question diagnostic checklist — Run any failing query through five yes/no checks (multi-document join required? order matters? causation chain? time span? why, not just what?) to score it as Class A, borderline, or Class B in under two minutes.
Four named failure patterns — Multi-hop relational failure, temporal sequencing failure, organizational context failure, and scale failure — each with a symptom description, a worked example, and a specific architectural fix.
Failure Classification Report template — A structured output artifact (query, class, failure patterns, root cause paragraph, recommended fix, references) that communicates a diagnosis clearly to engineers, architects, and non-technical stakeholders.
Architectural fix references — Each failure pattern maps directly to a companion skill (designing-hybrid-context-layers, temporal-reasoning-sleuth, synthesizing-institutional-knowledge) so diagnosis connects immediately to remediation.

Who Should Use This

Engineers and AI architects whose RAG pipeline is returning poor results and need to determine whether the problem is implementation quality (fixable with tuning) or architectural mismatch (requires a different retrieval approach).
Teams building agents over organizational knowledge bases — ADRs, incident reports, policy documents, vendor contracts — where some queries will always be relational or temporal in nature.
Technical leads evaluating whether to add a knowledge graph, timeline index, or hybrid retrieval layer and needing a principled basis for the recommendation rather than intuition.