Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    rag pipeline doctor

    1

    Diagnose and fix RAG retrieval failures, optimize chunking, and install evaluation loops for better AI accuracy.

    Free

    Security scanned
    rag pipeline doctor

    rag pipeline doctor

    Example session with this skill installed

    Our customer support bot is giving wrong answers for technical specs. I have the docs in Pinecone using standard chunking. Help me diagnose why it's missing specific part numbers.

    • Read your context and instructions
    • Compiled the rag pipeline doctor

    I have analyzed the reported failure regarding technical specification retrieval. Based on the symptom of missing specific part numbers, the primary diagnosis points to a

    Recall failure caused by a vocabulary gap in the vector-only search.

    RAG Diagnostic Report

    Failure LayerStatusObservationRecommended Fix
    CorpusPassSource documents contain the part numbers.N/A
    ChunkingWarningTables are being split, losing header context.Implement structure-aware splitting.
    RecallFAILVector search fails on exact alphanumeric strings.Enable Hybrid Search (BM25 + Vector).
    RankingWarningTop-k results are visually similar but irrelevant.Add a Cross-Encoder Reranker.
    GenerationPassModel tries to answer but lacks the right context.Improve grounding prompt.

    Technical Implementation Plan

    Hybrid Search: Integrate BM25 keyword matching alongside your Pinecone vector search. This ensures that exact part numbers (e.g., "TX-99-B") trigger a direct hit regardless of embedding distance.
    2.

    Structure-Aware Chunking: Re-index technical tables using Markdown-based splitting to keep row/column relationships intact.
    3.

    Reranking: Implement a cross-encoder/ms-marco-MiniLM-L-6-v2 model to re-score the top 50 retrieved chunks, passing only the top 5 to the LLM.

    Next steps

    • Provide a sample of 10 failed queries and their corresponding source snippets.
    • Update the retrieval function to fusion-score BM25 and Vector results.
    • Run a benchmark test to compare Recall@5 before and after these changes.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    RAG performance issues are rarely just "the model is hallucinating." Usually, the pipeline is diseased at the ingestion, retrieval, or ranking stage. This skill provides a structured diagnostic framework to isolate where data is being lost or misinterpreted and applies targeted fixes to the specific failure layer.

    What it does

    • Classifies failures by mapping symptoms to specific layers like corpus gaps, chunking errors, or recall mismatches.
    • Audits indexing strategy by evaluating chunk boundaries, metadata attachment, and embedding model suitability for your domain.
    • Optimizes retrieval by testing hybrid search (BM25 + vector) and cross-encoder reranking implementations.
    • Validates generation by checking citation accuracy and enforcing refusal paths for out-of-bounds queries.
    • Establishes evaluation loops by creating golden test sets and tracking recall@k and MRR metrics.

    How it works

    1. Target queries: Build a test set of 20 real user questions paired with golden answers and source documents.
    2. Diagnostics: Run the test set through the retrieval layer to calculate recall and identify the exact failure point.
    3. Anchor review: Audit chunking, embeddings, and reranking logic based on the diagnostic data.
    4. Citation check: Verify the model sticks to context and cites sources correctly.
    5. Evaluation: Install a versioned eval loop to measure the impact of every pipeline change.

    Frameworks & tools

    Works with any RAG stack including LangChain, LlamaIndex, Pinecone, Weaviate, and Chroma. Supports evaluation of embedding models, cross-encoders, and hybrid search configurations.

    Why this beats prompting it yourself

    Prompting for better answers won't fix a pipeline where the relevant context never reaches the LLM. This skill focuses on the engineering architecture, ensuring the retrieval mechanics are mathematically sound before adjusting the final prompt.

    Use cases

    • Fix chatbots that provide incorrect information despite having the right documents.
    • Improve search accuracy for technical documentation containing SKUs, error codes, or names.
    • Reduce hallucinations by enforcing strict context grounding and refusal paths.
    • Transition from prototype to production with a repeatable evaluation framework.

    Known limitations

    Requires a set of source documents and access to retrieval logs to perform diagnostics. Does not fix low-level infrastructure outages or network latency issues.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Requires source documents and retrieval logs. Focused on architectural pipeline fixes rather than hosting or infrastructure issues.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free to use

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 22 days ago

    • Free with an Agensi account

    Permissions

    No special permissions declared or detected

    Listed22 days ago

    What's inside

    Frequently Asked Questions

    The skill provides a complete diagnostic suite, including a structured framework for identifying retrieval gaps, an optimization guide for indexing/reranking, and a template for building automated evaluation loops (Recall@k and MRR).

    This skill is tool-agnostic and designed to integrate with major orchestration frameworks like LangChain and LlamaIndex, as well as vector databases such as Pinecone, Weaviate, and Chroma.

    Simply provide the skill with a representative set of source documents and access to your retrieval logs; it will then guide you through building a 20-query 'golden set' to begin the diagnostic process.

    Unlike prompt engineering which focuses on the LLM's 'voice,' this skill optimizes the underlying data retrieval architecture to ensure the model actually receives the correct information before it ever attempts to generate a response.

    Yes, the evaluation loops are designed to be versioned, allowing you to track how changes to your embedding models or chunking strategies impact accuracy over time as your corpus grows.

    New in LLM Ops

    Free