- Home
- Skills
- Data & Databases
- Leakage Check: Catch Data Leakage Before It Fakes Your Score
Leakage Check: Catch Data Leakage Before It Fakes Your Score
Makes your AI agent check a dataset or train/test split for target leakage and contamination before you trust a model. HIGH/MEDIUM/LOW findings with plain-language reasons. Tested Python, no API keys.
Free
Works with the AI Agents you already use
Leakage Check: Catch Data Leakage Before It Fakes Your Score
Example session with this skill installed
I trained a model on titanic3.csv to predict `survived` and got 97% accuracy. Is my data leaking?
- Read your context and instructions
- Compiled the leakage check
## Summary Your 97% is fake. Two columns leak the outcome: `boat` and `body` are only recorded after the ship sank. Without them, the same model scores an honest 79%. ## Leakage check **Target:** `survived` (classification) · 1309 rows · 13 features **Findings: 2 HIGH · 0 MEDIUM · 0 LOW** | Severity | Check | Column | What it means | |---|---|---|---| | HIGH | missing_predicts_target (also: target_proxy) | `boat` | Just knowing whether `boat` is empty predicts the target (AUC 0.971). Values that get filled in only after the outcome look exactly like this. | | HIGH | one_sided_value | `body` | When `body` is filled in, the target is `0` in 100% of 121 rows (overall only 62%). A value that exists for only one outcome is usually recorded after the outcome. | ### How well each column predicts the target on its own Out-of-fold AUC of a model that only sees one column (0.5 = useless, 1.0 = perfect). | Column | AUC | |---|---| | `boat` | 0.968 | | `sex` | 0.748 | | `fare` | 0.688 | | `ticket` | 0.677 | | `pclass` | 0.654 | ### Before / after (same model, 5-fold CV × 3) | | Columns used | Accuracy | ROC AUC | |---|---|---|---| | Before | 12, including `boat` and `body` | 0.974 | 0.993 | | After leakage-check | 10 (`boat`, `body` removed) | 0.787 | 0.847 | **Next steps** 1. Drop `boat` and `body`: neither is known at prediction time. 2. Split before any fitting: put imputers, scalers and encoders inside a scikit-learn `Pipeline` fit on training folds only. 3. Run the check again and report the honest score.
Connects securely to your tools. The creator never sees your data.
About this skill
Stop celebrating fake model scores. This skill makes your AI agent check for data leakage before you trust any model. **It finds:** - **Target leakage**: columns that contain the answer, or are only filled in after the outcome - **Train/test contamination**: shared rows, IDs, customers or time overlap between train and test - **Hidden red flags**: row order or IDs that predict the target, columns filled for only one outcome Every finding comes as HIGH / MEDIUM / LOW with a plain-language reason, plus a markdown and JSON report. **Real example:** on the Titanic dataset, an agent using every column scores 97% accuracy. Leakage Check flags `boat` and `body` as HIGH (both are only known after the ship sinks). The honest score is 79%, which is the one you can actually use. - Works with Claude Code and any agent that reads SKILL.md - One tested Python script: pandas, numpy, scikit-learn - No API keys, no network access, never changes your data - Open source (MIT), with tests and benchmarks: github.com/m-shahmeer-kz/dataskills
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Data & Databases.
Detects statistical patterns, not semantic context. Findings categorized as MEDIUM require manual domain verification. Logic relies on identifying correlations and overlaps, not causal reasoning.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free to use
No account required to browse
Trust & safety
Security scan
Security scan passed 2 days ago
- Free with an Agensi account
- Software
Python 3.9+pip