Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    RAG Eval Kit: Did Your Change Really Help?

    1

    There's a CI gate too, so a build fails when accuracy's lower confidence bound drops below your threshold.

    Free

    Security scanned
    RAG Eval Kit: Did Your Change Really Help?

    RAG Eval Kit: Did Your Change Really Help?

    Example session with this skill installed

    My RAG answers questions about a set of index rulebooks. I tried two changes on the same 126 labelled questions: a new answer prompt (v3 instead of v1) and a new answer model (qwen3 instead of qwen2.5). The runs are in this folder. Which change actually helped?

    • Read your context and instructions
    • Compiled the rag eval kit

    Summary

    Only the model swap is a real improvement. Prompt v3 scored 8.7 points lower than v1 (69.8% to 61.1%), but with 20 questions only v1 got right against 9 only v3 got right, the exact McNemar p is 0.0614:

    not distinguishable from noise at the 5% level. The model change raised accuracy from 69.8% to 81.7%, with 19 questions only the new model got right against 4:

    p = 0.0026, a real difference. It costs speed: median latency went from 2.75 s to 17.18 s.

    The runs, as scored

    prompt_v1 (126 questions)

    MetricScore95% interval
    Accuracy69.8% (88/126)61–77%
    Grounded accuracy55.6% (70/126)47–64%
    Answerable accuracy68.6% (81/118)60–76%
    Unanswerable refused87.5% (7/8)53–98%
    False refusals7.6% (9/118)4–14%
    Retrieval recall81.4% (96/118)73–87%
    Citation precision57.3% (71/124)48–66%
    Accuracy · paraphrase52.8% (19/36)37–68%
    Accuracy · unanswerable87.5% (7/8)53–98%
    Accuracy · untagged75.6% (62/82)65–84%

    Latency p50 2.18 s, p95 3.38 s.
    1 answer(s) contained a forbidden fact.

    prompt_v3 (126 questions)

    MetricScore95% interval
    Accuracy61.1% (77/126)52–69%
    Grounded accuracy51.6% (65/126)43–60%
    Answerable accuracy58.5% (69/118)50–67%
    Unanswerable refused100.0% (8/8)68–100%
    False refusals11.9% (14/118)7–19%
    Retrieval recall81.4% (96/118)73–87%
    Citation precision59.0% (69/117)50–68%
    Accuracy · paraphrase36.1% (13/36)22–52%
    Accuracy · unanswerable100.0% (8/8)68–100%
    Accuracy · untagged68.3% (56/82)58–77%

    Latency p50 2.02 s, p95 3.6 s.

    model_qwen2.5 (126 questions)

    MetricScore95% interval
    Accuracy69.8% (88/126)61–77%
    Grounded accuracy56.3% (71/126)48–65%
    Answerable accuracy69.5% (82/118)61–77%
    Unanswerable refused75.0% (6/8)41–93%
    False refusals8.5% (10/118)5–15%
    Retrieval recall80.5% (95/118)72–87%
    Citation precision52.9% (72/136)45–61%
    Accuracy · paraphrase52.8% (19/36)37–68%
    Accuracy · unanswerable75.0% (6/8)41–93%
    Accuracy · untagged76.8% (63/82)67–85%

    Latency p50 2.75 s, p95 4.19 s.
    2 answer(s) contained a forbidden fact.

    model_qwen3 (126 questions)

    MetricScore95% interval
    Accuracy81.7% (103/126)74–88%
    Grounded accuracy75.4% (95/126)67–82%
    Answerable accuracy80.5% (95/118)72–87%
    Unanswerable refused100.0% (8/8)68–100%
    False refusals11.9% (14/118)7–19%
    Retrieval recall82.2% (97/118)74–88%
    Citation precision69.0% (89/129)61–76%
    Accuracy · paraphrase55.6% (20/36)40–70%
    Accuracy · unanswerable100.0% (8/8)68–100%
    Accuracy · untagged91.5% (75/82)83–96%

    Latency p50 17.18 s, p95 23.6 s.

    The paired tests

    Prompt v1 (A) against prompt v3 (B):

    correct on 126 shared questions: A 69.8%  ->  B 61.1%  (-8.7 points)
    only A right: 20  p001 p002 p003 p005 p010 p022 p030 p033 p034 q002 q008 q016 q023 q026 q027 q034 q043 q047 q051 q055
    only B right: 9  p009 p012 p015 q009 q024 q050 q071 q073 u001
    exact McNemar p = 0.0614: not distinguishable from noise at the 5% level
    

    Old model (A) against new model (B):

    correct on 126 shared questions: A 69.8%  ->  B 81.7%  (+11.9 points)
    only A right: 4  p012 p028 p029 p035
    only B right: 19  p003 p008 p015 p022 p025 q009 q013 q023 q030 q045 q046 q050 q055 q061 q073 q080 q082 u001 u002
    exact McNemar p = 0.0026: a real difference at the 5% level
    

    What the numbers say

    The new model wins where it counts: answerable accuracy 69.5% (61-77%) to 80.5% (72-87%), grounded accuracy 56.3% to 75.4%, citation precision 52.9% to 69.0%, and every unanswerable trap refused (8/8, against 6/8).

    • Prompt v3 should not be adopted, but it is not proven worse either: its drop is largest on paraphrased questions (52.8% to 36.1%), and false refusals rose from 7.6% to 11.9%. A larger set could settle it.

    Retrieval did not change: recall stays between 80.5% and 82.2% in all four runs, so the gains come from how the model uses the pages it gets. The 18-20% of questions whose page is never retrieved remain a ceiling on accuracy.

    • Paraphrased questions remain the weak spot: 55.6% (40-70%) for the best run, against 91.5% on the rest.

    Read by hand (old model wrong, new model right): q009, the old model quoted the threshold for a neighbouring case (5%) instead of the one asked (15%), although the right page was retrieved; q023, it refused, and in that run the right page had not been retrieved, so that one is a retrieval miss rather than a model error. And the other way: on p012 the new model said the documents give no threshold, missing the expected 10%.

    Next steps

    • Keep the new model and prompt v1. Decide whether 17 s per answer is acceptable before shipping.
    • Work on paraphrased questions: query rewriting or retrieval tuned for them, measured on this dev set with compare.
    • Then score the held-out set once: python scripts/eval.py score heldout.jsonl pred_held.jsonl --heldout --system <your files>.
    • Add the gate to CI: python scripts/eval.py gate report.json --min-ci-low accuracy=72.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    What it does

    Most RAG "improvements" are noise. This skill tells you which are real. Your agent builds a labelled test set from your own documents, scores your system's answers without an LLM judge, and runs an exact paired test between two versions, so a new prompt, model, retriever or chunking gets a verdict you can defend.

    • Test set from your documents: text, Markdown, HTML, or PDFs (via pdftotext) cut into pages; pages drawn at random so questions aren't cherry-picked; paraphrased questions and unanswerable traps; every label checked against its page
    • Blind human review: a sheet with each label next to its source passage, reviewed without seeing any system's answers
    • Deterministic scoring: accuracy, grounded accuracy (right and citing the right page), refusals on unanswerable questions, false refusals, retrieval recall, citation precision, all with 95% intervals, broken down by question style
    • Is B really better than A? Exact McNemar test on the questions the two versions disagree on
    • Held-out discipline: a dev / held-out split with no shared pages, and a ledger that warns when the held-out set is reused on a new version
    • Fits your logs: converts JSONL, JSON or CSV output, LangChain-style source documents included, and matches by question text when there are no ids

    How it works

    Scripts do the counting and the statistics; your agent drafts the questions, runs the steps and explains the numbers. Three workflows: compare two runs you already have; build a test set from your documents (pages, random draw, draft, check, review, split); score a run and find where the errors come from (retrieval misses, wrong fact from the right page, false refusals). There's a CI gate too, so a build fails when accuracy's lower confidence bound drops below your threshold.

    Why you can trust it

    • No LLM judge: the same answers always get the same score, and every verdict traces back to the facts found or missing.
    • The agent may not call a difference real unless the paired test says so, and must quote intervals, never bare percentages.
    • Labels must pass a check against the source text before anything is scored, and unreviewed labels are reported as a draft.
    • Everything runs locally. Your documents and answers never leave your machine.

    Tested

    • 27 automated tests, all offline: the engine (scoring, normalisation, Wilson intervals, exact McNemar, label checks, held-out ledger) and the workflow (pages from text, Markdown, HTML and PDF page breaks, seeded sampling, the dev / held-out split, output conversion, the whole chain). All pass on Python 3.12 and 3.13.
    • Built on an open-source engine used to audit a 220-question RAG evaluation, where only 1 of 9 apparent improvements turned out to be statistically real.
    • Run in Claude Code.

    Known limits

    • It checks that the right facts are present, not that nothing wrong was added; forbidden facts cover known traps only, so read some answers.
    • Built for factual answers from documents, not summaries, open-ended writing or chat quality.
    • PDFs need converting to text first (pdftotext -layout keeps page numbers).
    • Intervals assume independent questions; near-duplicates make them too narrow.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Requires Python 3.10+. Checks that the right facts are present, not that nothing wrong was added; forbidden facts cover known traps only, so read some answers. Built for factual answers from documents, not summaries or chat quality. PDFs need converting to text first (pdftotext keeps page numbers). Intervals assume independent questions.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free to use

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 11 days ago

    • Free with an Agensi account

    Software
    Python 3.10+

    Permissions

    Terminal / Shell
    Read Files
    Write Files

    File Scopes

    rag-eval-kit/**
    **/*.jsonl
    **/*.json
    **/*.csv
    **/*.md

    No network access at all. Read Files: the documents, question sets and system outputs you name. Write Files: only the outputs you name (pages, questions, predictions, reports, review sheet).

    Listed11 days ago

    What's inside

    Frequently Asked Questions

    A judge model adds its own errors and changes its verdicts between runs. Here an answer is right when its key facts are present and no known wrong fact is, so the same answers always get the same score and every verdict can be checked.

    Yes, that is the main workflow. Your agent turns your documents into pages, draws pages at random, drafts questions with facts copied from each page, checks them against the source, and gives you a sheet to review the labels before anything is scored.

    Any that can log, per question, the answer and the sources with page numbers. It reads JSONL, JSON and CSV, understands LangChain-style source documents, and can match rows to questions by their text.

    About 60 give an accuracy within roughly 10 points, 250 within 5. Comparing two versions needs fewer, because the paired test only counts the questions where they disagree. The stats command prints the precision of your set.

    No. The scripts make no network calls and write only the files you name.

    The scoring engine is open source under MIT. The skill adds the workflow around it: building and reviewing the test set, converting your logs, and the rules your agent follows to report results honestly.

    New in LLM Ops

    Free