Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    RAG Failure Triage

    1

    Find out whether a bad RAG answer came from the query, retrieval, ranking, generation or the corpus, before you rewrite a single prompt.

    Free

    Security scanned
    RAG Failure Triage

    RAG Failure Triage

    Example session with this skill installed

    Triage this RAG failure. Question: "What is our refund window?" Query: "refund window". Top-3 chunks: [Doc A: Shipping info, Doc B: Pricing, Doc C: Contact]. Answer: "We don't have a refund window."

    • Read your context and instructions
    • Compiled the rag failure triage

    I have analyzed the artifacts provided for this RAG failure. The system failed to retrieve the relevant document despite the information being present in the corpus.

    RAG triage — case_001

    Question: What is our refund window?
    Query sent: "refund window"
    Top-k contained the answer: no (n/a)
    Answer used the context: yes (it correctly reflected the lack of info in the provided chunks)

    Diagnosis

    Layer: retrieval
    Cause: Lexical mismatch between the user query and the document vocabulary.

    Evidence

    • Chunk A (Score 0.82): Covers shipping timelines and carrier options.
    • Chunk B (Score 0.79): Lists subscription pricing tiers.
    • Chunk C (Score 0.75): General support contact email and hours.
    • Manual check: The source document uses the term "return period" instead of "refund window".

    Smallest next test

    Search the index using the specific phrase "return period". If the correct document appears in the top-3, the diagnosis of lexical mismatch is confirmed.

    If confirmed, fix

    • Implement query expansion to include synonyms like "return", "policy", and "reimbursement".
    • Enable hybrid search (BM25 + Vector) to better handle keyword matching for specific policy terms.

    Next steps

    1. Verify if the "return period" document is successfully indexed and not excluded by filters.
    2. Test the query "return period" manually in your vector DB console.
    3. Add "refund" and "return" to a synonym map in your search configuration.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Most RAG fixes are wasted because the wrong layer gets blamed: prompts get rewritten when the retriever never returned the answer, rerankers get added when the question was ambiguous. This is the ten-minute triage that finds the failure first.

    What you get

    • A decision tree that stops at the first true branch: coverage, retrieval miss, ranking, generation.
    • The evidence a real diagnosis needs (the query actually sent, the top-k chunks, the answer), and the finding to raise when it is missing.
    • Ten checks ordered by cost, from "is the document even indexed" to a fresh-index comparison.
    • A failure report template with the layer, the cause, the evidence and the smallest next test.

    How it works

    Take one failing case, print the retrieved chunks, read them by eye. Whether the answer is present, absent, or present but ignored tells you which layer to fix. Then change one thing and re-run the same case.

    Who it is for

    Anyone maintaining a retrieval system whose answers are sometimes wrong, incomplete or confidently invented.

    Limits

    It locates a failure; it does not measure how often it happens. One case tells you where to look, not a quality score.

    Free starter. For a 40-case failure taxonomy, retrieval metrics you can compute without labels, adversarial query generation and a before/after report, see RAG Diagnostics: https://www.agensi.io/skills/rag-diagnostics

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Locates single-case failures only. Does not measure aggregate quality or provide dataset-wide metrics. Requires raw retrieval logs (query, chunks, scores) to be effective.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free to use

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 2 days ago

    • Free with an Agensi account

    Requires

    Nothing to install.

    Permissions

    No special permissions declared or detected

    Listed2 days ago

    What's inside

    Frequently Asked Questions

    This skill provides a systematic diagnostic framework that uses a specific decision tree to isolate whether a failure occurred during query processing, vector retrieval, result ranking, or final generation, preventing you from wasting time on 'prompt engineering' when the data itself is missing.

    The skill is framework-agnostic and works with any stack including LangChain, LlamaIndex, or custom Python setups, as long as you can provide the core artifacts like the user query, top-k chunks, and retrieval scores.

    You receive the full automated triage logic, the decision-tree schemas for manual or programmatic audits, and a standardized reporting template to document fixes for your engineering team.

    This skill is designed for deep-dive, point-in-time triage of specific 'bad' responses; it is not a tool for high-volume batch processing or calculating aggregate metrics like Mean Reciprocal Rank (MRR).

    Setup is minimal; you simply need to ensure your RAG pipeline logs the retrieved context chunks and their associated similarity scores, as these artifacts are required for the triage process to begin.

    New in LLM Ops

    Free