data leakage detection lab

    1

    Inspects ML pipelines to identify and fix data leakage, contaminated features, and accidental hindsight.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    data leakage detection lab

    Example session with this skill installed

    Review this pipeline. I am using StandardScaler.fit(X) then train_test_split. My features include a rolling 30-day mean of the target variable. Is there leakage?

    • Read your context and instructions
    • Compiled the data leakage detection

    Leakage detected. 1) Preprocessing Leak: StandardScaler must be fit only on X_train to avoid test set statistics influencing training. 2) Future Leak: Rolling target means must be lagged or computed cumulatively to ensure future outcomes do not inform past predictions.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    The problem

    Machine learning models often report suspiciously high performance that collapses in production. This usually stems from data leakage, where information from the future or the test set accidentally contaminates the training process.

    What it does

    • Identifies future information leakage by auditing feature timestamps against label timestamps.
    • Detects preprocessing contamination where scalers, encoders, or imputers are fit on the full dataset before splitting.
    • Spots sample and group leakage where duplicate entities or near-duplicate records straddle the train-test boundary.
    • Exposes accidental hindsight by flagging features derived from revised, updated, or backfilled data.
    • Performs adversarial validation to determine if training and test sets are statistically distinguishable.

    Frameworks & tools

    Designed for Python-based data science stacks including scikit-learn, pandas, NumPy, and XGBoost/LightGBM. Works with Jupyter notebooks and automated ML pipelines.

    Why this beats prompting it yourself

    Standard LLMs often miss subtle leakage patterns like target encoding inside cross-validation or group-based dependencies. This skill uses specific diagnostic playbooks and adversarial tests to catch technical debt that manual reviews overlook.

    Use cases

    • Audit a Jupyter notebook before moving a model to a production environment.
    • Debug why a model achieved 0.99 AUC on real-world noisy data.
    • Verify that a temporal split correctly handles rolling windows and lag features.
    • Ensure multi-user datasets don't leak information via group identity.

    Known limitations

    Requires access to the data pipeline code and feature engineering logic. It cannot detect leakage in pre-compiled black-box models without seeing the training workflow.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Agents & Orchestration.

    Requires access to data pipeline code and feature engineering logic. Cannot audit pre-compiled black-box models without training workflow visibility.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 3 months ago

    Permissions

    No special permissions declared or detected

    Listed3 months ago

    What's inside

    Frequently Asked Questions

    It audits preprocessing logic to ensure scalers, encoders, and imputers are fit only on training data, preventing information bleed from the test set.

    Yes, it specifically looks for temporal dependencies, checking for future information leakage, lag feature errors, and incorrect rolling window calculations.

    It identifies when multiple records from the same entity (like a user or patient) exist in both training and test sets, which leads to overfit metrics.

    The skill requires access to the data pipeline code and feature engineering logic to perform a thorough audit. It cannot audit black-box models.

    New in Agents & Orchestration