Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Diagnosing RAG Failure Modes Skill

    1

    Diagnose silent RAG failures, hallucinated citations, and multi-hop reasoning gaps in retrieval pipelines.

    $9.99

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Diagnosing RAG Failure Modes Skill

    Diagnosing RAG Failure Modes Skill

    Example session with this skill installed

    I have a RAG failure. Query: 'How did the 2023 tax change affect Q4 revenue?' The answer says it decreased by 5%, but I can't find that in the retrieved chunks. Here is the trace.

    • Read your context and instructions
    • Compiled the diagnosing rag failure

    The diagnosis indicates a

    Grounding Failure combined with a

    Ranking Miss. While the model generated a specific figure, the retrieved evidence only discussed general tax policy without Q4 revenue data.

    RAG Diagnostic Report: Case #882

    HopSub-questionRequired evidenceRetrieved evidenceStatus
    12023 Tax Law changesSpecific provisionsPassage ID: doc_442Supported
    2Q4 Revenue figuresFinancial statementsNone foundMissing
    3Causal linkImpact analysisNone foundMissing

    Executive Diagnosis

    The system suffered a silent failure where the generator hallucinated a 5% decrease to satisfy the user's causal intent. The retrieval stage successfully found the tax law but failed to retrieve the corresponding Q4 financial report, likely because the financial document's embedding was not semantically similar to the "tax change" query.

    Pipeline Findings

    • Retrieval: Failed to fetch financial documents. Top-k (5) was filled with general tax commentary.
    • Ranking: A relevant summary document was ranked at position #12, falling outside the context window.
    • Generation: The model filled the evidentiary gap with prior knowledge/hallucination rather than abstaining.

    Recommended Fixes

    Hybrid Retrieval: Implement BM25 keyword search alongside vector search to capture specific entities like "Q4" and "Revenue".
    2.

    Query Expansion: Rewrite the query to explicitly include "financial results" when fiscal years or quarters are mentioned.
    3.

    Abstention Logic: Update the system prompt to require a "Data not available" response if Hop 2 evidence is missing.

    Next steps

    • Increase top_k to 20 and implement a Reranker to surface the financial summaries.
    • Add a validation step to check if retrieved chunks contain numerical values before generating financial answers.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    RAG systems often fail silently by returning fluent, authoritative answers that lack actual evidentiary support. This skill provides a systematic framework to decompose your retrieval-augmented generation pipeline and identify exactly where the logic breaks.

    What it does

    • Multi-hop analysis identifies missing intermediate facts or relationships in complex queries.
    • Stage-by-stage inspection isolates failures in query rewriting, vector retrieval, reranking, or context assembly.
    • Grounding verification maps individual answer claims to specific retrieved passages to detect hallucinations.
    • Causal auditing distinguishes between documented causality and inferred correlation in model responses.
    • Failure taxonomy classifies issues into 12 distinct categories like chunk-boundary errors or metadata filter misses.

    How it works

    1. Trace capture assembles the full lifecycle of a query including retrieved chunk IDs, similarity scores, and the final prompt.
    2. Evidence chain mapping defines the required facts and logical hops necessary to answer the user's intent.
    3. Pipeline traversal tests each stage sequentially to find the first point of failure.
    4. Fix recommendation generates targeted interventions for the specific root cause discovered.

    Frameworks & tools

    Works with any RAG stack including LangChain, LlamaIndex, or custom implementations using vector databases like Pinecone, Weaviate, or Milvus. It is model-agnostic and applies to any LLM-based generation.

    Why this beats prompting it yourself

    Generic prompts often miss "silent" failures where the answer looks correct but the citations are hollow. This skill enforces a rigorous evidence-chain methodology that forces the agent to prove every link in the reasoning path.

    Use cases

    • Debugging why a RAG system ignores relevant documents already present in the vector store.
    • Investigating "hallucinated" connections between two unrelated but correctly retrieved facts.
    • Optimizing chunking strategies for technical manuals where context is lost across headers.
    • Auditing citation accuracy to ensure links actually support the claims they are attached to.

    Known limitations

    Does not perform automated code execution to fix pipelines. Requires manual provision of traces or logs. Accuracy depends on the availability of retrieved context and metadata for inspection.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Requires manual provision of query traces and retrieved chunks. Does not automatically execute code fixes. Effectiveness limited if pipeline logs are incomplete or redacted.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 14 days ago

    Permissions

    No special permissions declared or detected

    Listed14 days ago

    What's inside

    Frequently Asked Questions

    This skill provides a systematic diagnostic framework and evidence-chain mapping tools to identify why your RAG system is hallucinating, missing relevant documents, or failing at complex multi-hop reasoning.

    The skill is model-agnostic and framework-independent; it integrates seamlessly with any stack using LangChain, LlamaIndex, or custom vector database implementations like Pinecone and Milvus.

    You receive the complete diagnostic framework, the 12-category failure taxonomy, and the evidence-chain mapping methodology, along with documentation on how to input your query traces for analysis.

    While it doesn't automatically rewrite your code, the skill provides a 'Fix Recommendation' output that generates targeted interventions and strategy adjustments based on the specific root cause discovered.

    To get started, you simply need to provide the skill with your query traces, including retrieved chunk IDs, similarity scores, and the final prompt generated by your system.

    Unlike basic prompts, this skill uses a multi-stage 'Pipeline Traversal' method to detect silent failures where a response sounds authoritative but lacks actual grounding in the retrieved context.

    New in LLM Ops