More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    rag retrieval eval set builder

    1

    Turn your docs into chunks plus grounded test questions, then score any retriever with recall@1/3/5/10 and MRR. Find out if RAG search actually finds the right chunk.

    $8.40

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    rag retrieval eval set builder

    rag retrieval eval set builder

    Example session with this skill installed

    Build a retrieval test set from the technical docs in /docs and run a score check against my current results in retrieved.jsonl. Set the minimum recall@5 to 80%.

    • Read your context and instructions
    • Compiled the rag retrieval eval

    I have processed the documents in /docs, generated a grounded QA test set, and evaluated your current retrieval performance. The system failed the performance gate as the Recall@5 was 74%, which is below your 80% requirement.

    RAG Retrieval Evaluation Report

    MetricValueStatus
    Total Test Queries150-
    Recall@10.52⚠️ Low
    Recall@30.68⚠️ Low
    Recall@50.74❌ Failed Gate (<0.80)
    Recall@100.82✅ Pass
    MRR (Mean Reciprocal Rank)0.61⚠️ Needs improvement

    Top Retrieval Failures

    • Query ID 42: "How do I configure mutual TLS?"
      • Expected Chunk: chunk_tls_09.md
      • Actual Rank: Not in top 10
      • Suggested Fix: Chunk size is too small; the context spans two headers. Increase chunk word cap.
    • Query ID 89: "Rate limit for tier 2 users"
      • Expected Chunk: limits_table_v2.md
      • Actual Rank: 7
      • Suggested Fix: Metadata filtering issue. The retriever is pulling v1 docs instead of v2.

    Next steps

    1. Review eval-qa.jsonl to see the generated questions for the failed chunks.
    2. Adjust the chunking word cap in the build script and regenerate the set.
    3. Update your vector store metadata filters to prioritize the latest documentation versions.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Most RAG bugs are retrieval bugs. Measure them before you tune the prompt.

    build splits Markdown/text docs into heading-aware chunks and generates grounded questions (definitions, numbers, how-tos, section questions) whose answer chunk is known. Run those questions through your retriever, save the returned chunk ids, and score reports recall@k, MRR and the questions that missed.

    What it does

    • Heading-aware chunking with a word cap
    • Auto-generated questions with known gold chunk ids
    • recall@1/3/5/10 + MRR, list of misses
    • --min-recall5 gate for CI

    Who it's for

    Developers building RAG chatbots, doc search or support assistants.

    Limits

    Template questions are simpler than real user queries - add real ones too. Not legal/financial advice.

    What's included

    • SKILL.md: workflow, commands, how to explain findings
    • scripts/: the tool (stdlib Python)
    • references/sources.md: specs and guidance cited
    • examples/: sample inputs used in the demo
    • tests/: unit tests (Python 3.8 / 3.9 / 3.13)

    Standard-library Python only. No API keys, no network calls, no installs.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Offline / local files only - does not fetch live URLs or call APIs.
    Template questions are simpler than real user queries - add real ones too.
    Scores retrieval only, not answer quality.
    Not legal/financial advice.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 16 days ago

    Permissions

    Terminal / Shell
    Read Files
    Write Files

    File Scopes

    rag-retrieval-eval-set-builder/**

    Fully offline. No subprocess, no network calls, no credential access.

    Listed16 days ago

    What's inside

    Frequently Asked Questions

    No - it is retriever-agnostic. You run the questions through your own stack and give it the returned ids.

    They are template-based and grounded in the text; review and edit qa.jsonl for important sets.

    Markdown and plain text folders.

    New in LLM Ops