RAG Pipeline Production Auditor
Scan your RAG codebase and eval set, then get a scorecard and ranked fixes for chunking, retrieval, grounding, guardrails and cost.
Free
Works with the AI Agents you already use
Claude Code
Cursor
Codex CLI
Muse
Dots+22 moreClaude CodeCursorCodex CLIMuseDots
OpenClaw
Manus
Grok
Grok Bot+18 agentsRAG Pipeline Production Auditor
Example session with this skill installed
Audit our support bot before launch: repo in rag_app/, eval questions in eval.jsonl.
- Read your context and instructions
- Compiled the rag pipeline production
(This is the scanner's real output on the bundled example app, shortened.)
# RAG pipeline audit
Files scanned: 2 | areas with evidence: 2/17 | risky settings: 5 | gaps: 15
Risky settings
| medium | answer.py:5 | Passing 30 chunks (k=30) | rerank down to ~3-8 |
| medium | answer.py:8 | temperature = 0.9 for answer generation | use 0-0.3 for grounded Q&A |
| medium | answer.py:7 | user input + context in one f-string | delimit context, mark untrusted |
| medium | ingest.py:4 | Large chunk_size (4000 chars) | compare smaller sizes on eval set |
| low | ingest.py:4 | chunk_overlap = 0 | 10-20% overlap / structure-aware |
No evidence found: grounding_prompt (high), eval (high), guardrails_injection (high), citations, reranking, access_control, cost_controls, observability ...
Eval set: 3 rows | 2 with expected sources | 0 unanswerable | 1 duplicate
The agent then writes the scorecard (for example Generation & grounding 0/2: no "answer only from context" rule and no citations). The fix plan is ordered: build a 50+ question golden set with source ids and unanswerables → grounding prompt and citations → reranker over k=30 → max_tokens and usage logging. Each step names the metric to compare before and after.
Connects securely to your tools. The creator never sees your data.
About this skill
A pre-launch review for retrieval-augmented generation apps, based on your actual code rather than a generic questionnaire.
- Maps your pipeline. loader → splitter → embeddings → store → retriever → reranker → prompt → LLM → citations and guardrails, with file paths.
- Runs a read-only scanner (Python stdlib, no model calls). It finds evidence across 17 areas with file:line: chunking, chunk metadata, embedding pinning and versioning, hybrid search, metadata filters, reranking, relevance thresholds, grounding prompt, citations, evals, prompt-injection and PII guardrails, access control, cost controls, tracing, timeouts.
- Flags risky settings. Very large or zero-overlap chunks, k over 20 without a reranker, high temperature, user input and context concatenated into one f-string, and hard-coded API keys (reported redacted, never echoed).
- Checks your eval set (JSONL/CSV): size, reference answers, expected source ids, unanswerable questions and duplicates.
- Scores 8 areas 0–2 with a 37-item checklist, and gives an ordered fix plan where every fix is paired with the metric that proves it (recall@k, MRR, faithfulness, refusal rate, cost per request).
Built by an engineer who ships production RAG, including a solo-built consumer AI app.
What it won't do: call your models or vector DB, send documents anywhere, or edit code. It also won't claim accuracy gains. You measure them on your eval set.
How to use
Skills on Agensi run inside your AI agent. Copy the prompt, paste it into your agent, and it connects to Agensi and installs this skill for you. If a payment is needed, your agent shows it to you first.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill passes our trust checks below.
Security scanned
Passed our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free forever
No account required to browse
Trust & safety
Security scanned
Verified clean 4 days ago
- Free to download with an account
Requires