Works with the AI Agents you already use
AI Eval Test Suite Quality Gate Skill Pro
Audit LLM evaluations and test suites for gameable rubrics, data leakage, and reliability issues.
$14
AI Eval Test Suite Quality Gate Skill Pro
Example session with this skill installed
Audit my new RAG eval suite. I've attached the judge_rubric.md, the test_cases.json, and the eval_script.py. We are seeing high scores but frequent hallucinations in prod.
- Read your context and instructions
- Compiled the ai eval test
I have completed the audit of your RAG evaluation suite. While the scoring logic is sound, a critical loophole in the judge rubric and significant edge-case gaps render the current 94% pass rate untrustworthy.
Overall Verdict: REVISE
Audit Summary
| ID | Severity | Category | Evidence | Confidence |
|---|---|---|---|---|
| FIND-01 | Critical | Judge Rubric | judge_rubric.md: Lines 12-18 | High |
| FIND-02 | High | Edge-Case Coverage | test_cases.json: Missing null/empty context | High |
| FIND-03 | Medium | Flakiness | eval_script.py: Temperature 0.7 on Judge | Medium |
Findings & Remediation
[FIND-01] Critical — Verbosity Bias in Judge Rubric
Observation: The rubric rewards "comprehensive and detailed" answers without a penalty for irrelevant information.
Risk: The model can achieve a '5/5' score by hallucinating extra details as long as it maintains a professional tone, explaining the gap between eval scores and production failures.
Recommended fix: Add a conciseness constraint and a "Negative Constraints" section to the rubric that penalizes information not present in the source context.
[FIND-02] High — Missing Adversarial Contexts
- Observation: The dataset only contains samples where the answer exists in the context.
- Risk: The system's ability to "refuse to answer" when context is missing or irrelevant is completely untested.
Recommended fix: Add 15-20 "No-Answer" test cases where the retrieval step is mocked to return empty or unrelated documents.
Next steps
- Update
judge_rubric.mdwith the provided conciseness constraints. - Inject "Unanswerable" cases into
test_cases.jsonto measure refusal accuracy. - Pin the Judge LLM temperature to 0 in
eval_script.pyto reduce scoring variance.
Connects securely to your tools. The creator never sees your data.
About this skill
Your LLM evaluation suite might be lying to you. High scores often hide gameable rubrics, data leakage, and brittle metrics that fail in production. This skill audits your AI test pipelines to ensure your quality gates are actually trustworthy.
What it does
- Judge Rubric Audit identifies vague scoring criteria, position bias, and loopholes in LLM-as-a-judge prompts.
- Leakage Detection finds instances where ground-truth labels or reference answers have contaminated the model context or training splits.
- Edge-Case Assessment maps your test cases against adversarial inputs, boundary values, and failure modes like timeouts or partial outputs.
- Reliability Review flags nondeterminism caused by unpinned model versions, lack of seeds, or race conditions in the test harness.
- Scoring Verification validates aggregation logic to ensure averages don't mask critical failures in specific data slices.
How it works
- Inventory: Provide your system prompts, judge rubrics, evaluation datasets, and test scripts.
- Analysis: The agent audits the artifacts against seven dimensions including validity, flakiness, and reproducibility.
- Verdict: Receive a PASS, REVISE, or FAIL verdict based on evidence-backed findings.
- Remediation: Follow the generated plan to fix critical leakage or gameable scoring logic.
Frameworks & tools
Works with any evaluation stack including LangSmith, Promptfoo, DeepEval, or custom Python/Node.js test harnesses. It analyzes YAML, JSON, and code-based test definitions.
Why this beats prompting it yourself
Generic prompts often miss subtle data contamination or judge biases that skew results. This skill applies a systematic audit framework that separates observed defects from hypothetical risks and provides a formal severity/confidence matrix.
Use cases
- Auditing a new RAG evaluation pipeline before a production release.
- Investigating why model benchmarks don't match real-world user feedback.
- Strengthening judge rubrics to prevent models from "gaming" high scores through verbosity.
- Verifying dataset split hygiene to prevent over-optimistic performance metrics.
Known limitations
Does not execute live code or perform network-level security testing. Review is limited to the artifacts and datasets explicitly provided or accessible in the environment.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
Limited to artifact analysis; does not execute code or live model calls. Requires access to rubrics and datasets for high-confidence verdicts. Scope is restricted to provided files.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 20 days ago
Permissions
No special permissions declared or detected