rag eval harness

    1

    Build a statistically rigorous evaluation harness for RAG apps with golden sets, retrieval metrics, and error analysis.

    $12

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    rag eval harness

    Example session with this skill installed

    I've made some changes to my RAG pipeline's chunking strategy and added a reranker. I need to know if it's actually better than the old version. Can you build an eval harness for this?

    • Read your context and instructions
    • Compiled the rag eval harness

    I have initialized the evaluation harness in the eval/ and scripts/ directories. I analyzed your retriever.py and app.py to identify the retrieval and generation entry points.

    RAG Evaluation Harness

    I have generated a starter golden set of 30 questions. Please review eval/questions.jsonl to ensure the gold_ids match the expected source chunks before running the full benchmark.

    
    # eval/run_eval.py snippet
    import json
    import time
    from your_app import rag_pipeline
    
    def run_benchmark(questions_path, output_name):
        results = []
        with open(questions_path, 'r') as f:
            for line in f:
                q = json.loads(line)
                start = time.time()
                # Captures retrieved_ids, answer, and citations
                response = rag_pipeline(q['question']) 
                
                results.append({
                    "id": q['id'],
                    "question": q['question'],
                    "answer": response.text,
                    "retrieved_ids": response.chunk_ids,
                    "gold_ids": q['gold_ids'],
                    "latency_s": time.time() - start
                })
        # Save to eval/results/<output_name>.jsonl
    

    Initial Baseline Metrics (N=30)

    MetricMean95% Conf. Interval
    hit@30.83[0.71, 0.94]
    MRR0.76[0.65, 0.88]
    Faithfulness0.88[0.82, 0.94]
    Context Precision0.72[0.60, 0.84]

    Next steps

    1. Review eval/questions.jsonl and mark at least 10 questions as split: "test".
    2. Run python scripts/metrics.py eval/results/new_chunking.jsonl --baseline eval/results/baseline.jsonl to see the delta.
    3. Check the "Error Analysis" section in the generated report to see the top 5 failure cases.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    RAG applications often ship with zero measurement or anecdotal evidence that fails in production. This skill builds a repeatable evaluation harness directly in your repository to provide defensible metrics on retrieval quality and generation accuracy. It moves you from guessing if a change helped to knowing with statistical confidence.

    What it does

    • Golden set generation creates a structured questions.jsonl covering single facts, paraphrases, and multi-chunk queries.
    • Retrieval benchmarking calculates hit@k and Mean Reciprocal Rank (MRR) to verify your embedding and reranking logic.
    • Generation auditing measures faithfulness, context precision, and answer relevance using LLM-as-a-judge patterns.
    • Cost estimation predicts token spend for evaluation runs before execution to prevent surprise bills.
    • Statistical aggregation computes mean, standard deviation, and 95 percent confidence intervals to distinguish signal from noise.
    • Error diagnosis maps failure symptoms to root causes like chunk size, retrieval misses, or model hallucination.

    How it works

    1. Analyze pipeline by identifying your entry points, retrieval functions, and current configuration.
    2. Bootstrap golden set using synthetic generation followed by a mandatory human review phase to ensure label quality.
    3. Execute runs via a dedicated script that captures inputs, outputs, and intermediate retrieval states.
    4. Compute and compare results against baselines to see if changes to your stack actually improved performance.

    Frameworks & tools

    The harness is built with Python, utilizing JSONL for data storage and basic statistical libraries for bootstrap intervals. It is designed to wrap any RAG stack regardless of the specific vector database or LLM provider.

    Why this beats prompting it yourself

    Generic prompts cannot handle the nuances of RAG evaluation like data leakage or judge bias. This skill enforces rigorous practices like held-out test splits, paired comparisons, and reporting N-size with confidence intervals to ensure your numbers are actually meaningful.

    Use cases

    • Benchmark a new embedding model or reranker against your current production baseline.
    • Detect regressions in faithfulness after updating your system prompt or LLM version.
    • Build a defensible quality report for stakeholders using statistically sound metrics.
    • Create a CI smoke test to block pull requests that degrade retrieval performance.

    Known limitations

    Requires a manual review of at least a third of generated questions to ensure accuracy. Small question sets under 20 items will result in wide confidence intervals.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Requires manual review of generated questions. Small N-sizes (under 20) yield wide confidence intervals. Judge agreement must be hand-checked on 20 items.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 14 days ago

    Software
    Python

    Permissions

    Terminal / Shell
    Read Files
    Write Files
    Listed14 days ago

    What's inside

    Frequently Asked Questions

    This skill moves beyond anecdotal testing by building a repeatable framework that measures specific metrics like Hit@k, MRR, and faithfulness, allowing you to prove with statistical confidence that your RAG changes actually improve performance.

    The harness is framework-agnostic and designed to wrap around any existing RAG pipeline; as long as your system is built in Python, you can integrate these benchmarking scripts regardless of your specific vector database or LLM provider.

    You receive a complete evaluation kit for your repository, including scripts for golden set generation, benchmarking runners, statistical analysis modules, and templates for error diagnosis and cost estimation.

    While the harness automates the bulk of the work, it requires a human-in-the-loop phase to review a portion of the generated golden set to ensure high-quality ground truth labels, which is critical for defensible metrics.

    Absolutely. The harness is designed to compute mean, standard deviation, and 95% confidence intervals, making it ideal for creating high-stakes quality reports for stakeholders or acting as a CI/CD gate for production deployments.

    New in LLM Ops