Works with the AI Agents you already use
RAG Eval Harness: evals that check their own metrics
Turns "the chatbot seems fine" into numbers you can put in a README and fail a build on.
Free
RAG Eval Harness: evals that check their own metrics
Example session with this skill installed
evaluate my RAG app
- Read your context and instructions
- Compiled the rag eval harness
Evaluated the Advocate Remuneration Assistant, a legal RAG app over Kenya's Advocates (Remuneration) Order and ~200 court rulings.
What I did
- Inspected the app: retriever
RagPipeline.retrieve()passes k = 7 chunks with stable ids; generator is Groqgpt-oss-20b; answers cite[chunk_id: ...], so the[N]citation check does not apply. - Wrote
evals/rag_adapter.pyaround the app's realanswer()call, passing the judge the same context the model sees, including the fee calculator's verified result. - Built a 30-question golden set from the project's hand-reviewed questions: 20 lookups with gold chunk ids, 4 calculations, 6 out-of-scope (20%). Frozen at sha256
df359f2d5677b5c9. - Chose
gpt-5.4-minias judge: different from the generator and from the model that wrote the questions. - Ran at the production k:
python evals/run_eval.py --golden evals/golden_set.jsonl --k 7
evals/results.md
| Slice | hit | mrr | faithfulness | relevancy | citation | refused | false_refusal |
|---|---|---|---|---|---|---|---|
| all | 0.9 | 0.545 | 0.97 | 0.938 | None | 1.0 | 0.042 |
| calculation (4) | None | None | 1.0 | 1.0 | None | None | 0.0 |
| lookup (20) | 0.9 | 0.545 | 0.963 | 0.925 | None | None | 0.05 |
| out_of_scope (6) | None | None | None | None | None | 1.0 | None |
Checking the metrics before trusting them
- A first run scored calculation faithfulness 0.25 although all 4 fees were correct: the adapter had not passed the calculator's result to the judge. Fixed; the 4 now score 1.0.
- Negative controls (clean/broken pairs) failed 2 of 4 on their first run: DeepEval's faithfulness by default counts claims the context neither supports nor contradicts as faithful, so an invented fee scored 1.0. Unsupported claims now count against faithfulness, refusals are judged by the refusal checks instead, and all 4 pairs pass on consecutive runs.
- The one false refusal ("Not covered" on an answerable question) gets no faithfulness score; it is flagged as a false refusal, with relevancy 0.0.
- Two runs of the same 20 lookups differed by about 0.06 in faithfulness: the noise floor.
Next steps
- Treat these scores as in-sample: the questions come from a set the app was later tuned against. Confirm any improvement on newly written questions.
- Retrieval found the gold chunk for 18 of 20 lookups; read the two misses before changing chunking or k.
- Add the CI gate from
templates/rag-eval.yml, with thresholds about 5 points below this baseline, plus the negative-control step.
Connects securely to your tools. The creator never sees your data.
About this skill
Turns "the chatbot seems fine" into numbers you can put in a README and fail a build on. Works with any RAG app that can return its answer and the chunks it retrieved.
What it does when you ask your agent to evaluate a RAG app:
- Reads your code to find the retriever, the generator and the citation format.
- Writes a thin adapter around your real pipeline, never a re-implementation.
- Helps you build a golden set from the source documents, with gold chunk ids and out-of-scope questions.
- Sets up a judge model (Groq free tier, local Ollama, or OpenAI), never the same small model as the generator.
- Runs hit@k, MRR, faithfulness, answer relevancy, citation validity, refusal rate and false-refusal rate.
- Checks the metrics themselves: can each one fail, does it move when it should, does it reward refusals, and what is the noise floor.
- Compares changes as paired runs with per-question flips, one change at a time.
- Optionally adds a GitHub Actions gate: retrieval-only metrics on every PR, the judged run on main.
Why step 6 matters: auditing a real legal RAG assistant scored on 236 pre-registered questions found a retrieval metric that passed by construction on 72% of questions, a judge that gave wrong "not covered" refusals a perfect 5/5, and two router bugs. With those fixed, re-ranking raised accuracy from 89.4% to 93.6% and cut wrong refusals from 10 to 1 (exploratory). Full write-up in the GitHub README.
Each run writes evals/results.md and evals/results.jsonl, so you can re-score a run without regenerating answers. MIT licensed.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
Requires stable chunk IDs for retrieval metrics; the skill adds them first if your chunks have none. Judge-scored metrics cost API calls and vary between runs: run the same configuration twice to see the noise floor, and use a judge of 8B+ parameters. Refusal detection is a keyword check, and citation checking covers [N]-style markers only. The harness itself runs in Python. You still write and verify the golden set's reference answers yourself.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free to use
No account required to browse
Trust & safety
Security scan
Security scan passed 6 days ago
- Free with an Agensi account
- Software
Python
Git