Works with the AI Agents you already use
RAG Reality Check
Audits what a RAG system really does, not what its docs or scores claim: the executed code path, silent failures, retrieval measured on its own, and the blind spots of metrics like faithfulness.
$12
RAG Reality Check
Example session with this skill installed
Audit our RAG assistant in ./app before we present it to the client. The README says hybrid search with an HNSW index and a faithfulness of 0.9.
- Read your context and instructions
- Compiled the rag reality check
An audit report: claims table (e.g. "HNSW index: false, exact search on the executed path, index only reached by dead code"), the executed path of one query, silent failures ranked worst first, hit@k / recall@k / MRR per probe type on a stated number of probes, the cross-table showing questions scored high on faithfulness although retrieval missed, failure modes, security notes and ranked recommendations.
Connects securely to your tools. The creator never sees your data.
About this skill
A RAG pipeline can score 0.9 on faithfulness while retrieving the wrong documents. Faithfulness checks that the answer sticks to the context it was given, not that the context was the right one, nor that the answer is correct. This skill audits a RAG system the way a sceptical reviewer would:
- Claims vs reality. Every claim from the docs (hybrid search, vector index, tool use, scores) ends as verified, false or unverified, with file:line or an observed request payload as evidence.
- Executed path. Follows one real query through the code actually called, and spots dead code the docs describe as active.
- Silent failures. A Python AST scanner ranks exception handlers that swallow errors, print instead of logging or return empty values: the kind that turns an outage into "no content found".
- Retrieval measured on its own. A probe set covering answerable, out-of-scope, cross-lingual, near-topic and multi-hop questions; a script computes hit@k, recall@k and MRR per probe type, detects a retriever with no relevance threshold, and crosses retrieval with your answer metric to show where the metric is blind.
- Security quick pass and a structured report, with every figure tied to its command and probe count.
Built from a real audit. Standard-library Python only; works with any RAG stack (you export retrieved ids per question).
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
The silent-failure scanner reads Python only; other languages are checked by hand following the same patterns. Retrieval figures depend on the probe set: expected sources must be written by someone who knows the corpus. You need to export retrieved ids per question from your system (guidance included). Model-judged metrics are not computed by the skill: it crosses the ones you already have with retrieval.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 17 days ago
- Software
Python 3.8+
Permissions
File Scopes
No network access in the scripts. Exporting results may require running your own system, in a throwaway branch.