More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    ML Pipeline Auditor: Is Your Model's Accuracy Real?

    1

    This skill makes your AI agent review machine-learning code and Jupyter notebooks like a senior data scientist.

    $12

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    ML Pipeline Auditor: Is Your Model's Accuracy Real?

    ML Pipeline Auditor: Is Your Model's Accuracy Real?

    Example session with this skill installed

    My Titanic notebook leaky_titanic.py gets F1 0.99, which feels too good. Audit it with the ML Pipeline Auditor: is the score real? The data is titanic3.csv in the same folder.

    • Read your context and instructions
    • Compiled the ml pipeline auditor
    • Generated the data export

    ML Pipeline Audit: titanic survival notebook

    Verdict: the reported F1 of 0.99 cannot be trusted. Three problems let the model see the test answers, so the score measures leakage, not skill.

    Score: 0/100 (grade F) · 1 file (leaky_titanic.py) + data check on titanic3.csv · Reported metric: F1 0.986, "AUC" 0.985

    AreaHighMediumLow
    Leakage240
    Validation101
    Metrics010
    Reproducibility000
    Deep learning000

    Fix first

    1. Remove the full-data target encoding (ticket_rate, cabin_rate, home.dest_rate), or rebuild it with TargetEncoder inside a Pipeline.
    2. Split before SMOTE, and put SMOTE inside an imblearn Pipeline.
    3. Tune max_depth with cross-validation on the training data, then use the test set once.

    After these three fixes the expected score is 100/100. Re-run training: honest F1 will be clearly lower (see the end).

    Findings

    1. [HIGH] MLA106 Target encoding on the full dataset (lines 21-23)

    What: df.groupby(col)["survived"].transform("mean") for ticket, cabin, home.dest, computed on all 1,309 rows before the split.

    Why it matters here: 713 passengers (54%) have a ticket number nobody else has. For them ticket_rate

    is their own survival label (0 or 1). The forest simply reads it. This alone explains most of the 0.99.

    • Fix:
    
    # before
    df["ticket_rate"] = df.groupby("ticket")["survived"].transform("mean")
    
    # after: learned inside each training fold with cross-fitting
    from sklearn.preprocessing import TargetEncoder
    ("te", TargetEncoder(cv=KFold(5, shuffle=True, random_state=SEED)), ["ticket", "cabin", "home.dest"])
    

    2. [HIGH] MLA104 SMOTE before the split (line 31)

    • What: SMOTE().fit_resample(X, y) on all rows, then train_test_split.

    Why it matters here: synthetic survivors are interpolated from real passengers, so near-copies of training rows land in the test set (and the test set no longer has the real 38% survival rate).

    Fix: split first; Pipeline([("pre", pre), ("smote", SMOTE()), ("rf", RandomForestClassifier())]) from imblearn.pipeline.

    3. [HIGH] MLA201 Test set used to choose max_depth (lines 35-40)

    • What: the loop keeps the depth with the best test F1, then reports test F1 again.

    Fix: GridSearchCV(pipe, {"rf__max_depth": [3, 5, 8, 12, None]}, cv=StratifiedKFold(5, shuffle=True, random_state=SEED), scoring="f1") on the training data only.

    4. [MEDIUM] MLA301 "AUC" computed from hard labels (line 43)

    • roc_auc_score(y_test, pred) with pred = model.predict(...). Use model.predict_proba(X_test)[:, 1].

    5. [MEDIUM] MLA108 Fill values from all rows (lines 17, 18, 23)

    • age mean and fare median include the test rows; line 23 even fills with the mean of the

    target. Use SimpleImputer inside the Pipeline. Small effect for age/fare; line 23 is part of finding 1.

    6. [MEDIUM] MLA102 StandardScaler on all rows (line 30)

    • Real leak in principle, but

    harmless here: random forests do not depend on feature scale. Fix it for correctness (inside the Pipeline), lowest priority.

    Checked and dismissed

    • MLA206 (no stratify, line 32): valid hint. Add stratify=y when you split before SMOTE.

    Data split check

    split_check.py on a random split of titanic3.csv: no duplicate rows across the split, class shares within 2.4 points,

    but boat alone predicts survival (category purity 0.98). Good news: this notebook does not use boat or body (both are only known after the sinking). Keep it that way.

    What was reviewed manually

    • Post-outcome features: boat, body correctly excluded. home.dest is known before the trip: fine as a raw feature, leaky only through the target encoding.
    • Evaluation unit: passengers who travelled together share tickets. A group split by ticket would be stricter; mention it if the model must generalise to new families.
    • Metric: F1 is reasonable for a 38% positive class. Also report ROC-AUC from probabilities.

    Re-measured after the fixes

    Same model, all fixes applied (fixed_titanic.py), same 20 random splits:

    F1 (mean ± sd, 20 splits)Audit score
    Original notebook0.991 ± 0.0050/100
    After fixes0.749 ± 0.027 (ROC-AUC 0.871)100/100

    Generated with ML Pipeline Auditor 1.0.1: static analysis + data check + manual review. Your training code was not run during the audit; the re-measurement above was run separately.

    titanic3.csv

    CSV · data export

    Generated

    Example file from a real run - the skill writes it into your workspace.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Is your model's accuracy real? This skill makes your AI agent review machine-learning code and Jupyter notebooks like a senior data scientist. It finds the mistakes that make results look better than they are, proves each one with file and line, and gives you the exact fix.

    What your agent does

    1. Runs a static analyser over your .py scripts and .ipynb notebooks (26 rules, cell-aware, notebook magics handled).
    2. Opens every finding and decides: confirmed, intended, or false alarm. No blind copying.
    3. Works through a manual checklist for what code scanning cannot see.
    4. Checks your real train/test files when you have them.
    5. Writes ML_AUDIT_REPORT.md: verdict, 0-100 score, findings ordered by impact on your metric, before/after code fixes, a "fix first" list.
    6. With your OK, applies the fixes (usually one Pipeline) and re-scores.

    What it catches

    • Leakage: feature selection, target encoding, SMOTE, scaling or imputation fitted before the split or outside cross-validation; target column left in X; random split of time series; the same patient/user on both sides.
    • Validation: test set used to tune hyperparameters or for early stopping; model trained on test data; training-only evaluation; threshold tuned on the test set; no stratification.
    • Metrics: ROC-AUC / log loss computed from hard labels; swapped metric(y_pred, y_true); accuracy as the only metric.
    • Reproducibility: missing seeds; notebook cells run out of order or skipped; unpinned dependencies.
    • Deep learning: missing optimizer.zero_grad(), missing model.eval(), softmax/sigmoid before a loss that expects logits.
    • Your data (split checker): duplicate and near-duplicate rows across train/test, shared IDs or groups, test rows older than training rows, single features that predict the target, drift.
    • Manual checklist: post-outcome features, preprocessing hidden in SQL, wrong evaluation unit, metric vs business goal, and LLM/RAG evaluation contamination (few-shot examples from the test set, prompts tuned on the test set, LLM-as-judge bias).

    Tested, with real numbers

    • Real data demo: a typical Titanic notebook reports F1 0.991 (± 0.005 over 20 random splits). After the skill's fixes the honest F1 is 0.749 (± 0.027). Audit score 0/100 vs 100/100.
    • Real-world code: run on 447 files from scikit-learn's examples, Hands-On ML, Microsoft ML-For-Beginners and the Python Data Science Handbook; every high/medium finding checked by hand. On the two projects held out until the analyser was frozen, 15 of 16 findings were correct or debatable; the one false alarm was fixed in v1.0.1. It found real issues there, including a swapped classification_report(ypred, ytest) in the Python Data Science Handbook and a scaler fitted on all data that scikit-learn's own comment calls "cheating a bit".
    • Labelled cases: 31/31 planted problems caught, 0 of 15 clean pipelines flagged. 70 automated tests.

    For teams

    • --fail-on high blocks pull requests that add leakage.
    • SARIF 2.1.0 output for GitHub code-scanning alerts; Markdown and JSON reports.
    • Ready-made GitHub Actions, GitLab CI and pre-commit snippets.
    • # mlaudit: ignore[MLA102] reason for accepted exceptions.

    Good to know

    • Private and local. Nothing is uploaded. The analyser reads code; it never runs your training.
    • Nothing to install. The code analyser is pure Python standard library (tested on Python 3.9 to 3.14). The data checker needs pandas and numpy.
    • Honest limits. Static analysis can miss leaks that live in SQL, other repositories or the data itself; that is what the data checker and the manual checklist are for.

    Full results, sample reports and the demo: https://m-shahmeer-kz.github.io/ml-pipeline-auditor/

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Code Quality & Review.

    Static analysis reads Python code only: leaks that happen in SQL, in another repository or inside the data itself need the data split check and the manual checklist. Rules are tuned for scikit-learn, imbalanced-learn, pandas, XGBoost/LightGBM, Keras and PyTorch idioms. The data split check needs pandas and numpy.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 1 day ago

    Software
    Python 3.9+
    Optional
    pandasnumpy (only for the data split check)

    Runs fully on your machine. If your agent receives the skill as text, it saves the two scripts to .mlaudit/scripts/ in your project first, then runs them.

    Permissions

    Terminal / Shell
    Read Files
    Write Files

    File Scopes

    **/*.py
    **/*.ipynb
    requirements.txt
    ML_AUDIT_REPORT.md
    .mlaudit/**

    Terminal: runs two local Python scripts (code analyser, data split check). Read files: your .py/.ipynb code, requirements.txt and optional train/test data. Write files: the audit report, scratch files in .mlaudit/ (deleted at the end) and, only with your OK, the code fixes. No network; nothing is uploaded.

    Listed1 day ago

    What's inside

    Frequently Asked Questions

    ML-specific mistakes that inflate results: feature selection, target encoding, SMOTE, scaling or imputation fitted before the split or outside cross-validation; the test set used for tuning or early stopping; AUC computed from hard labels; swapped metric arguments; random splits of time series; notebooks run out of order; missing zero_grad()/eval() in PyTorch. 26 rules in total, plus a data split check and a manual checklist.

    Yes. It reads .ipynb files cell by cell, handles %magics and !shell lines, reports findings as 'cell N (In [n]) line M' so you can find them at a glance, and also flags notebooks whose cells were run out of order or skipped.

    Python 3.9 or newer. The code analyser uses only the standard library, so there is nothing to install. The optional train/test data check needs pandas and numpy.

    No. The analyser only reads your code, and the data check only reads the files you point it to. Nothing leaves your machine; it needs no network access.

    On real code it had not seen before (Microsoft ML-For-Beginners and the Python Data Science Handbook), 15 of 16 findings were correct or debatable; the one false alarm was fixed in v1.0.1. It caught 31/31 planted problems and flagged 0 of 15 clean pipelines. Your agent also double-checks every finding before reporting it.

    Yes. Run it with --fail-on high to block pull requests that add leakage, and with --format sarif to show findings as GitHub code-scanning alerts. Ready-made GitHub Actions, GitLab CI and pre-commit snippets are included.

    New in Code Quality & Review