Leakage Check: Catch Data Leakage Before It Fakes Your Score

    1

    Makes your AI agent check a dataset or train/test split for target leakage and contamination before you trust a model. HIGH/MEDIUM/LOW findings with plain-language reasons. Tested Python, no API keys.

    Free

    Security scanned

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Leakage Check: Catch Data Leakage Before It Fakes Your Score

    Example session with this skill installed

    I trained a model on titanic3.csv to predict `survived` and got 97% accuracy. Is my data leaking?

    • Read your context and instructions
    • Compiled the leakage check

    ## Summary Your 97% is fake. Two columns leak the outcome: `boat` and `body` are only recorded after the ship sank. Without them, the same model scores an honest 79%. ## Leakage check **Target:** `survived` (classification) · 1309 rows · 13 features **Findings: 2 HIGH · 0 MEDIUM · 0 LOW** | Severity | Check | Column | What it means | |---|---|---|---| | HIGH | missing_predicts_target (also: target_proxy) | `boat` | Just knowing whether `boat` is empty predicts the target (AUC 0.971). Values that get filled in only after the outcome look exactly like this. | | HIGH | one_sided_value | `body` | When `body` is filled in, the target is `0` in 100% of 121 rows (overall only 62%). A value that exists for only one outcome is usually recorded after the outcome. | ### How well each column predicts the target on its own Out-of-fold AUC of a model that only sees one column (0.5 = useless, 1.0 = perfect). | Column | AUC | |---|---| | `boat` | 0.968 | | `sex` | 0.748 | | `fare` | 0.688 | | `ticket` | 0.677 | | `pclass` | 0.654 | ### Before / after (same model, 5-fold CV × 3) | | Columns used | Accuracy | ROC AUC | |---|---|---|---| | Before | 12, including `boat` and `body` | 0.974 | 0.993 | | After leakage-check | 10 (`boat`, `body` removed) | 0.787 | 0.847 | **Next steps** 1. Drop `boat` and `body`: neither is known at prediction time. 2. Split before any fitting: put imputers, scalers and encoders inside a scikit-learn `Pipeline` fit on training folds only. 3. Run the check again and report the honest score.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Stop celebrating fake model scores. This skill makes your AI agent check for data leakage before you trust any model. **It finds:** - **Target leakage**: columns that contain the answer, or are only filled in after the outcome - **Train/test contamination**: shared rows, IDs, customers or time overlap between train and test - **Hidden red flags**: row order or IDs that predict the target, columns filled for only one outcome Every finding comes as HIGH / MEDIUM / LOW with a plain-language reason, plus a markdown and JSON report. **Real example:** on the Titanic dataset, an agent using every column scores 97% accuracy. Leakage Check flags `boat` and `body` as HIGH (both are only known after the ship sinks). The honest score is 79%, which is the one you can actually use. - Works with Claude Code and any agent that reads SKILL.md - One tested Python script: pandas, numpy, scikit-learn - No API keys, no network access, never changes your data - Open source (MIT), with tests and benchmarks: github.com/m-shahmeer-kz/dataskills

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Data & Databases.

    Detects statistical patterns, not semantic context. Findings categorized as MEDIUM require manual domain verification. Logic relies on identifying correlations and overlaps, not causal reasoning.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free to use

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 2 days ago

    • Free with an Agensi account

    Software
    Python 3.9+pip

    Permissions

    Terminal / Shell
    Read Files
    Write Files
    Listed2 days ago

    What's inside

    Frequently Asked Questions

    This skill automates the detection of 'too good to be true' features, such as columns that contain the answer or data collected after the event, as well as hidden overlaps between your training and testing sets.

    The skill is a Python-based utility that works with standard data science libraries like Pandas and Scikit-learn, making it compatible with any AI agent or workflow that supports Python execution environments.

    The SKILL.md instructions, the tested leakage_check.py script, requirements.txt and a short install note. Each run writes a markdown report and a JSON report. The skill is free and open source (MIT).

    While the tool identifies statistical anomalies and suspicious correlations, you must define the 'prediction moment' to help the skill distinguish between legitimate predictive features and data that wouldn't actually be available at inference time.

    It runs locally with pandas and scikit-learn, so speed depends on your machine. Very large or very wide tables take longer and need more memory; you can run it on a sample of rows first.

    By flagging HIGH and MEDIUM severity risks, the skill prevents you from deploying models with inflated metrics that would otherwise fail immediately when exposed to real-world, unseen data.

    New in Data & Databases

    Free