More screenshots
Works with the AI Agents you already use
Eval Leakage Check — Is Your Test Set Contaminated?
Reads your eval JSONL and reports duplicate cases, answers sitting in the prompt, test inputs that also appear in train, and label imbalance before you quote a score.
Free
Eval Leakage Check — Is Your Test Set Contaminated?
Example session with this skill installed
python3 scripts/leak_scan.py cases.jsonl --train train.jsonl --fail-on high
{"id":"e04","split":"test","input":"Customer asks how to reset access…",
"expected":"verify identity then send a reset link for account 4"}
- Read your context and instructions
- Compiled the eval leakage check
eval-leakage-check 53 cases test 41 train 12
findings 7 high 4 medium 2 low 1
duplicates 9 near-pairs 3 answer leaks 3 split overlap 4
HIGH E4 exact duplicate input 9 duplicate cases across 9 inputs
HIGH E6 answer in input 3 cases contain the expected answer
HIGH E7 test input in train 4 test inputs also appear in train
FAIL findings at or above 'high'
Connects securely to your tools. The creator never sees your data.
About this skill
A score is only as honest as the test set. If the answer is in the prompt, the case was copied from train, or the same question appears three times, the number measures memory and duplication rather than capability. This reads the files and shows the leakage that is visible without a model.
It checks exact duplicates by normalised input, near-duplicates by shared five-word shingles, answers that appear verbatim in the prompt, test inputs that also appear in train, and labels shared across splits. It reports empty inputs and targets, duplicate ids, very short placeholders, and a label distribution where one answer covers more than 90% of the test set.
- Exact and near-duplicate detection with the affected case ids
- Answer-in-input detection that ignores normalisation noise
- Test/train overlap for both exact and 80%-similar cases
- Duplicate ids and empty targets, which silently break a pass rate
- Label-imbalance check against the majority-class baseline
- Exit 1 at a severity threshold, so contamination fails CI
The shipped failing file is 53 cases with 9 duplicates, 3 answers embedded in the prompt, 4 test inputs copied from train and one empty target. The clean file is 36 cases with no findings.
Good to know: this checks exact and five-gram overlap only. Paraphrases, semantic familiarity and model memorisation are not detected. A clean result means "not detected by these checks", not "uncontaminated", because the training data is not in the file. N-gram matches can be incidental on templated text. Split overlap is only checked when a train/dev split or a separate train file is provided.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
Detects exact and 5-gram overlaps only. No semantic or paraphrase detection. Only checks provided local files; cannot account for model pre-training data. Requires JSONL format.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free to use
No account required to browse
Trust & safety
Security scan
Security scan passed 1 day ago
- Free with an Agensi account