Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    AI Agent Knowledge Retrieval Tester

    1

    Systematically audit RAG agent performance, detect hallucinations, and validate knowledge base grounding.

    $19.99

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    AI Agent Knowledge Retrieval Tester

    AI Agent Knowledge Retrieval Tester

    Example session with this skill installed

    I've provided a technical manual and an agent's answer about 'Error Code 502'. The agent says it's a battery issue, but I suspect that's wrong. Please evaluate the grounding and citations.

    • Read your context and instructions
    • Compiled the ai agent knowledge

    I have evaluated the agent's response against the provided technical manual. The agent incorrectly identified the error code, leading to a grounding failure.

    RAG Evaluation Report: Error Code 502

    DimensionScoreAssessment
    Retrieval Relevance5/5Correct manual section for 'System Errors' was retrieved.
    Evidence Coverage5/5The source explicitly defines Error 502 on page 12.
    Answer Faithfulness0/5Agent claimed 'Battery Failure'; source states 'Network Timeout'.
    Citation Support1/5Citation points to the correct page, but the claim contradicts the text.
    Abstention Behavior0/5Agent provided a false answer instead of citing the source correctly.

    Failure Diagnosis:

    • Contradicted: The answer conflicts with the supplied source documentation.

    Hallucination: The agent added unsupported details regarding battery hardware not present in the retrieved text.

    Priority Recommendations:

    • Priority: Critical.
    • Observed Issue: Agent ignores retrieved text in favor of internal weights.
    • Recommended Action: Implement a 'strict grounding' prompt step or an evidence-verification layer before output.
    • Expected Improvement: Reduction in factual contradictions.

    Next steps

    1. Review the temperature settings on the generation LLM to reduce creative drift.
    2. Update the system prompt to explicitly forbid answering if the retrieved text does not contain the specific error code definition.
    3. Add the 'Error Code 502' case to your regression test pack.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Developing RAG systems often leads to silent failures like hallucinations, poor retrieval ranking, or ungrounded answers. This skill provides a systematic framework for evaluating how AI agents use supplied knowledge bases to ensure factual accuracy and citation integrity.

    What it does

    • Knowledge auditing analyzes documents to identify coverage gaps and potential retrieval risks.
    • Test set generation creates balanced suites of fact lookups, multi-hop queries, and out-of-scope edge cases.
    • Grounding evaluation compares agent responses against retrieved passages to detect unsupported claims or hallucinations.
    • Retrieval diagnostics assesses passage relevance, context preservation, and chunking effectiveness.
    • Performance scoring applies a consistent 0-5 scale across dimensions like faithfulness, completeness, and abstention behavior.
    • Regression planning produces reusable test packs with clear pass/fail criteria for continuous improvement.

    How it works

    1. Scope confirmation defines the evaluation objectives and identifies available evidence like logs or source excerpts.
    2. Analysis execution inspects retrieved passages and generated answers against the ground-truth knowledge base.
    3. Failure classification assigns evidence-based labels such as "relevant source not retrieved" or "answer adds unsupported details."
    4. Report generation delivers a structured audit including scores, failure patterns, and prioritized technical recommendations.

    Frameworks & tools

    This tool works with any RAG architecture, vector database, or LLM orchestration framework. It requires text-based inputs such as knowledge base excerpts, retrieval logs (JSON/text), or agent interaction histories.

    Why this beats prompting it yourself

    Generic prompts often fail to distinguish between retrieval failures and generation hallucinations. This skill enforces a rigorous multi-step inspection that separates factual correctness from writing fluency, preventing the "hallucination of accuracy" in fluent but ungrounded agent responses.

    Use cases

    • Auditing a customer support bot to ensure it only answers from official manuals.
    • Debugging why an agent fails to synthesize information across multiple documents.
    • Validating citation accuracy in legal or technical documentation assistants.
    • Testing RAG abstention logic for queries that fall outside the knowledge base.

    Known limitations

    Cannot directly connect to live vector databases or production telemetry. Evaluations are limited to the specific excerpts and logs provided in the session.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Cannot connect to live databases or production telemetry. Evaluation is restricted to provided excerpts and logs. Does not tune RAG pipelines or change embeddings directly.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 15 days ago

    Permissions

    No special permissions declared or detected

    Listed15 days ago

    What's inside

    Frequently Asked Questions

    This skill is framework-agnostic and works with any RAG setup, including LangChain, LlamaIndex, or custom-built agents, provided you can supply text-based interaction logs and knowledge base excerpts.

    It solves the problem of "silent failures" where an agent sounds confident but is actually hallucinating or failing to retrieve the correct data, a distinction generic testing often misses.

    You receive the full auditing framework, including automated test set generation logic, the 0-5 performance scoring system, and templates for failure classification and regression planning.

    No, the skill is designed for offline evaluation using exported logs and excerpts to ensure data privacy and avoid complex API integrations with your production environment.

    The skill includes a dedicated "abstention logic" test to verify that your agent correctly stays silent or admits lack of knowledge when a query falls outside your provided documentation.

    New in LLM Ops