Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    RAG Eval Harness: evals that check their own metrics

    1

    Turns "the chatbot seems fine" into numbers you can put in a README and fail a build on.

    Free

    Security scanned
    RAG Eval Harness: evals that check their own metrics

    RAG Eval Harness: evals that check their own metrics

    Example session with this skill installed

    evaluate my RAG app

    • Read your context and instructions
    • Compiled the rag eval harness

    Evaluated the Advocate Remuneration Assistant, a legal RAG app over Kenya's Advocates (Remuneration) Order and ~200 court rulings.

    What I did

    1. Inspected the app: retriever RagPipeline.retrieve() passes k = 7 chunks with stable ids; generator is Groq gpt-oss-20b; answers cite [chunk_id: ...], so the [N] citation check does not apply.
    2. Wrote evals/rag_adapter.py around the app's real answer() call, passing the judge the same context the model sees, including the fee calculator's verified result.
    3. Built a 30-question golden set from the project's hand-reviewed questions: 20 lookups with gold chunk ids, 4 calculations, 6 out-of-scope (20%). Frozen at sha256 df359f2d5677b5c9.
    4. Chose gpt-5.4-mini as judge: different from the generator and from the model that wrote the questions.
    5. Ran at the production k: python evals/run_eval.py --golden evals/golden_set.jsonl --k 7

    evals/results.md

    Slicehitmrrfaithfulnessrelevancycitationrefusedfalse_refusal
    all0.90.5450.970.938None1.00.042
    calculation (4)NoneNone1.01.0NoneNone0.0
    lookup (20)0.90.5450.9630.925NoneNone0.05
    out_of_scope (6)NoneNoneNoneNoneNone1.0None

    Checking the metrics before trusting them

    • A first run scored calculation faithfulness 0.25 although all 4 fees were correct: the adapter had not passed the calculator's result to the judge. Fixed; the 4 now score 1.0.
    • Negative controls (clean/broken pairs) failed 2 of 4 on their first run: DeepEval's faithfulness by default counts claims the context neither supports nor contradicts as faithful, so an invented fee scored 1.0. Unsupported claims now count against faithfulness, refusals are judged by the refusal checks instead, and all 4 pairs pass on consecutive runs.
    • The one false refusal ("Not covered" on an answerable question) gets no faithfulness score; it is flagged as a false refusal, with relevancy 0.0.
    • Two runs of the same 20 lookups differed by about 0.06 in faithfulness: the noise floor.

    Next steps

    • Treat these scores as in-sample: the questions come from a set the app was later tuned against. Confirm any improvement on newly written questions.
    • Retrieval found the gold chunk for 18 of 20 lookups; read the two misses before changing chunking or k.
    • Add the CI gate from templates/rag-eval.yml, with thresholds about 5 points below this baseline, plus the negative-control step.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Turns "the chatbot seems fine" into numbers you can put in a README and fail a build on. Works with any RAG app that can return its answer and the chunks it retrieved.

    What it does when you ask your agent to evaluate a RAG app:

    1. Reads your code to find the retriever, the generator and the citation format.
    2. Writes a thin adapter around your real pipeline, never a re-implementation.
    3. Helps you build a golden set from the source documents, with gold chunk ids and out-of-scope questions.
    4. Sets up a judge model (Groq free tier, local Ollama, or OpenAI), never the same small model as the generator.
    5. Runs hit@k, MRR, faithfulness, answer relevancy, citation validity, refusal rate and false-refusal rate.
    6. Checks the metrics themselves: can each one fail, does it move when it should, does it reward refusals, and what is the noise floor.
    7. Compares changes as paired runs with per-question flips, one change at a time.
    8. Optionally adds a GitHub Actions gate: retrieval-only metrics on every PR, the judged run on main.

    Why step 6 matters: auditing a real legal RAG assistant scored on 236 pre-registered questions found a retrieval metric that passed by construction on 72% of questions, a judge that gave wrong "not covered" refusals a perfect 5/5, and two router bugs. With those fixed, re-ranking raised accuracy from 89.4% to 93.6% and cut wrong refusals from 10 to 1 (exploratory). Full write-up in the GitHub README.

    Each run writes evals/results.md and evals/results.jsonl, so you can re-score a run without regenerating answers. MIT licensed.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Requires stable chunk IDs for retrieval metrics; the skill adds them first if your chunks have none. Judge-scored metrics cost API calls and vary between runs: run the same configuration twice to see the noise floor, and use a judge of 8B+ parameters. Refusal detection is a keyword check, and citation checking covers [N]-style markers only. The harness itself runs in Python. You still write and verify the golden set's reference answers yourself.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free to use

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 6 days ago

    • Free with an Agensi account

    Software
    PythonGit

    Permissions

    Terminal / Shell
    Read Files
    Write Files
    Network Access
    Environment Variables
    Listed6 days ago

    What's inside

    Frequently Asked Questions

    This skill replaces subjective 'vibe checks' with hard metrics like Hit@k and MRR, allowing you to pinpoint whether a failure is happening in your vector search (retrieval) or your LLM output (generation).

    The harness is compatible with any Python-based RAG stack, including those built on LangChain or LlamaIndex, provided your retriever can return stable unique identifiers for data chunks.

    The skill includes SKILL.md (the workflow your agent follows), the evaluation script run_eval.py, a golden-set template and a GitHub Actions workflow. Your agent writes the adapter for your own app when you run it.

    Yes. The judge can be a local model through Ollama, Groq's free hosted tier, or OpenAI. Use a different model from the one your app generates with, and preferably 8B or larger, since smaller judges often return malformed JSON.

    No. The skill creates an evals/ directory with an adapter around your existing pipeline, so your application logic stays untouched, unless your chunks have no stable IDs, in which case it adds them first.

    Faithfulness judges score "I don't know" as fully supported, so a model that refuses more looks better. The harness reports a false-refusal rate on answerable questions next to faithfulness and next to the out-of-scope refusal rate. In v1.1, refusals are not given a faithfulness score at all: they are judged by the refusal and false-refusal checks, so in the demo the wrong "Not covered" shows up as a false refusal with relevancy 0.0.

    New in LLM Ops

    Free