More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Eval Leakage Check — Is Your Test Set Contaminated?

    1

    Reads your eval JSONL and reports duplicate cases, answers sitting in the prompt, test inputs that also appear in train, and label imbalance before you quote a score.

    Free

    Security scanned
    Eval Leakage Check — Is Your Test Set Contaminated?

    Eval Leakage Check — Is Your Test Set Contaminated?

    Example session with this skill installed

    python3 scripts/leak_scan.py cases.jsonl --train train.jsonl --fail-on high

    {"id":"e04","split":"test","input":"Customer asks how to reset access…",
    "expected":"verify identity then send a reset link for account 4"}

    • Read your context and instructions
    • Compiled the eval leakage check

    eval-leakage-check 53 cases test 41 train 12
    findings 7 high 4 medium 2 low 1
    duplicates 9 near-pairs 3 answer leaks 3 split overlap 4

    HIGH E4 exact duplicate input 9 duplicate cases across 9 inputs
    HIGH E6 answer in input 3 cases contain the expected answer
    HIGH E7 test input in train 4 test inputs also appear in train

    FAIL findings at or above 'high'

    Connects securely to your tools. The creator never sees your data.

    About this skill

    A score is only as honest as the test set. If the answer is in the prompt, the case was copied from train, or the same question appears three times, the number measures memory and duplication rather than capability. This reads the files and shows the leakage that is visible without a model.

    It checks exact duplicates by normalised input, near-duplicates by shared five-word shingles, answers that appear verbatim in the prompt, test inputs that also appear in train, and labels shared across splits. It reports empty inputs and targets, duplicate ids, very short placeholders, and a label distribution where one answer covers more than 90% of the test set.

    • Exact and near-duplicate detection with the affected case ids
    • Answer-in-input detection that ignores normalisation noise
    • Test/train overlap for both exact and 80%-similar cases
    • Duplicate ids and empty targets, which silently break a pass rate
    • Label-imbalance check against the majority-class baseline
    • Exit 1 at a severity threshold, so contamination fails CI

    The shipped failing file is 53 cases with 9 duplicates, 3 answers embedded in the prompt, 4 test inputs copied from train and one empty target. The clean file is 36 cases with no findings.

    Good to know: this checks exact and five-gram overlap only. Paraphrases, semantic familiarity and model memorisation are not detected. A clean result means "not detected by these checks", not "uncontaminated", because the training data is not in the file. N-gram matches can be incidental on templated text. Split overlap is only checked when a train/dev split or a separate train file is provided.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Detects exact and 5-gram overlaps only. No semantic or paraphrase detection. Only checks provided local files; cannot account for model pre-training data. Requires JSONL format.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free to use

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 1 day ago

    • Free with an Agensi account

    Permissions

    Terminal / Shell
    Read Files
    Write Files
    Network Access
    Listed1 day ago

    What's inside

    Frequently Asked Questions

    This skill detects 'data contamination,' where answers are accidentally included in prompts or test data exists in the training set, causing artificially inflated accuracy scores. It ensures your evaluation results are legitimate and not the result of the model simply 'memorizing' the answers.

    No, this tool uses a model-free, n-gram matching approach that runs entirely on your local machine. This makes it significantly faster than using an LLM and ensures you don't incur any API token costs or send sensitive data to third-party providers.

    The skill is a standalone Python utility (3.10+) that accepts standard JSONL files. It can be integrated into any AI agent workflow, CI/CD pipeline, or development environment that supports Python scripts and file-based data.

    You receive the full Python source code for the leakage detection engine, documentation on integration, and a sample JSONL dataset to test the tool's functionality immediately upon purchase.

    The tool is highly effective at finding exact matches and near-duplicates (80% 5-gram overlap), but it does not detect 'semantic' leakage where the meaning is the same but the wording is entirely different. It focuses on deterministic data overlaps rather than conceptual paraphrasing.

    Yes, the tool is designed for automation; it generates a detailed Markdown report for human review and returns specific exit codes that allow you to automatically block builds or deployments if leakage thresholds are exceeded.

    New in LLM Ops

    Free