- Home
- Skills
- Code Quality & Review
- ML Pipeline Auditor: Is Your Model's Accuracy Real?
More screenshots
Works with the AI Agents you already use
ML Pipeline Auditor: Is Your Model's Accuracy Real?
This skill makes your AI agent review machine-learning code and Jupyter notebooks like a senior data scientist.
$12
ML Pipeline Auditor: Is Your Model's Accuracy Real?
Example session with this skill installed
My Titanic notebook leaky_titanic.py gets F1 0.99, which feels too good. Audit it with the ML Pipeline Auditor: is the score real? The data is titanic3.csv in the same folder.
- Read your context and instructions
- Compiled the ml pipeline auditor
- Generated the data export
ML Pipeline Audit: titanic survival notebook
Verdict: the reported F1 of 0.99 cannot be trusted. Three problems let the model see the test answers, so the score measures leakage, not skill.
Score: 0/100 (grade F) · 1 file (leaky_titanic.py) + data check on titanic3.csv · Reported metric: F1 0.986, "AUC" 0.985
| Area | High | Medium | Low |
|---|---|---|---|
| Leakage | 2 | 4 | 0 |
| Validation | 1 | 0 | 1 |
| Metrics | 0 | 1 | 0 |
| Reproducibility | 0 | 0 | 0 |
| Deep learning | 0 | 0 | 0 |
Fix first
- Remove the full-data target encoding (
ticket_rate,cabin_rate,home.dest_rate), or rebuild it withTargetEncoderinside a Pipeline. - Split before SMOTE, and put SMOTE inside an
imblearnPipeline. - Tune
max_depthwith cross-validation on the training data, then use the test set once.
After these three fixes the expected score is 100/100. Re-run training: honest F1 will be clearly lower (see the end).
Findings
1. [HIGH] MLA106 Target encoding on the full dataset (lines 21-23)
What: df.groupby(col)["survived"].transform("mean") for ticket, cabin, home.dest, computed on all 1,309 rows before the split.
Why it matters here: 713 passengers (54%) have a ticket number nobody else has. For them ticket_rate
is their own survival label (0 or 1). The forest simply reads it. This alone explains most of the 0.99.
- Fix:
# before
df["ticket_rate"] = df.groupby("ticket")["survived"].transform("mean")
# after: learned inside each training fold with cross-fitting
from sklearn.preprocessing import TargetEncoder
("te", TargetEncoder(cv=KFold(5, shuffle=True, random_state=SEED)), ["ticket", "cabin", "home.dest"])
2. [HIGH] MLA104 SMOTE before the split (line 31)
- What:
SMOTE().fit_resample(X, y)on all rows, thentrain_test_split.
Why it matters here: synthetic survivors are interpolated from real passengers, so near-copies of training rows land in the test set (and the test set no longer has the real 38% survival rate).
Fix: split first; Pipeline([("pre", pre), ("smote", SMOTE()), ("rf", RandomForestClassifier())]) from imblearn.pipeline.
3. [HIGH] MLA201 Test set used to choose max_depth (lines 35-40)
- What: the loop keeps the depth with the best test F1, then reports test F1 again.
Fix: GridSearchCV(pipe, {"rf__max_depth": [3, 5, 8, 12, None]}, cv=StratifiedKFold(5, shuffle=True, random_state=SEED), scoring="f1") on the training data only.
4. [MEDIUM] MLA301 "AUC" computed from hard labels (line 43)
roc_auc_score(y_test, pred)withpred = model.predict(...). Usemodel.predict_proba(X_test)[:, 1].
5. [MEDIUM] MLA108 Fill values from all rows (lines 17, 18, 23)
agemean andfaremedian include the test rows; line 23 even fills with the mean of the
target. Use SimpleImputer inside the Pipeline. Small effect for age/fare; line 23 is part of finding 1.
6. [MEDIUM] MLA102 StandardScaler on all rows (line 30)
- Real leak in principle, but
harmless here: random forests do not depend on feature scale. Fix it for correctness (inside the Pipeline), lowest priority.
Checked and dismissed
- MLA206 (no
stratify, line 32): valid hint. Addstratify=ywhen you split before SMOTE.
Data split check
split_check.py on a random split of titanic3.csv: no duplicate rows across the split, class shares within 2.4 points,
but boat alone predicts survival (category purity 0.98). Good news: this notebook does not use boat or body (both are only known after the sinking). Keep it that way.
What was reviewed manually
- Post-outcome features:
boat,bodycorrectly excluded.home.destis known before the trip: fine as a raw feature, leaky only through the target encoding. - Evaluation unit: passengers who travelled together share tickets. A group split by
ticketwould be stricter; mention it if the model must generalise to new families. - Metric: F1 is reasonable for a 38% positive class. Also report ROC-AUC from probabilities.
Re-measured after the fixes
Same model, all fixes applied (fixed_titanic.py), same 20 random splits:
| F1 (mean ± sd, 20 splits) | Audit score | |
|---|---|---|
| Original notebook | 0.991 ± 0.005 | 0/100 |
| After fixes | 0.749 ± 0.027 (ROC-AUC 0.871) | 100/100 |
Generated with ML Pipeline Auditor 1.0.1: static analysis + data check + manual review. Your training code was not run during the audit; the re-measurement above was run separately.
titanic3.csv
CSV · data export
Example file from a real run - the skill writes it into your workspace.
Connects securely to your tools. The creator never sees your data.
About this skill
Is your model's accuracy real? This skill makes your AI agent review machine-learning code and Jupyter notebooks like a senior data scientist. It finds the mistakes that make results look better than they are, proves each one with file and line, and gives you the exact fix.
What your agent does
- Runs a static analyser over your
.pyscripts and.ipynbnotebooks (26 rules, cell-aware, notebook magics handled). - Opens every finding and decides: confirmed, intended, or false alarm. No blind copying.
- Works through a manual checklist for what code scanning cannot see.
- Checks your real train/test files when you have them.
- Writes
ML_AUDIT_REPORT.md: verdict, 0-100 score, findings ordered by impact on your metric, before/after code fixes, a "fix first" list. - With your OK, applies the fixes (usually one
Pipeline) and re-scores.
What it catches
- Leakage: feature selection, target encoding, SMOTE, scaling or imputation fitted before the split or outside cross-validation; target column left in X; random split of time series; the same patient/user on both sides.
- Validation: test set used to tune hyperparameters or for early stopping; model trained on test data; training-only evaluation; threshold tuned on the test set; no stratification.
- Metrics: ROC-AUC / log loss computed from hard labels; swapped
metric(y_pred, y_true); accuracy as the only metric. - Reproducibility: missing seeds; notebook cells run out of order or skipped; unpinned dependencies.
- Deep learning: missing
optimizer.zero_grad(), missingmodel.eval(), softmax/sigmoid before a loss that expects logits. - Your data (split checker): duplicate and near-duplicate rows across train/test, shared IDs or groups, test rows older than training rows, single features that predict the target, drift.
- Manual checklist: post-outcome features, preprocessing hidden in SQL, wrong evaluation unit, metric vs business goal, and LLM/RAG evaluation contamination (few-shot examples from the test set, prompts tuned on the test set, LLM-as-judge bias).
Tested, with real numbers
- Real data demo: a typical Titanic notebook reports F1 0.991 (± 0.005 over 20 random splits). After the skill's fixes the honest F1 is 0.749 (± 0.027). Audit score 0/100 vs 100/100.
- Real-world code: run on 447 files from scikit-learn's examples, Hands-On ML, Microsoft ML-For-Beginners and the Python Data Science Handbook; every high/medium finding checked by hand. On the two projects held out until the analyser was frozen, 15 of 16 findings were correct or debatable; the one false alarm was fixed in v1.0.1. It found real issues there, including a swapped
classification_report(ypred, ytest)in the Python Data Science Handbook and a scaler fitted on all data that scikit-learn's own comment calls "cheating a bit". - Labelled cases: 31/31 planted problems caught, 0 of 15 clean pipelines flagged. 70 automated tests.
For teams
--fail-on highblocks pull requests that add leakage.- SARIF 2.1.0 output for GitHub code-scanning alerts; Markdown and JSON reports.
- Ready-made GitHub Actions, GitLab CI and pre-commit snippets.
# mlaudit: ignore[MLA102] reasonfor accepted exceptions.
Good to know
- Private and local. Nothing is uploaded. The analyser reads code; it never runs your training.
- Nothing to install. The code analyser is pure Python standard library (tested on Python 3.9 to 3.14). The data checker needs pandas and numpy.
- Honest limits. Static analysis can miss leaks that live in SQL, other repositories or the data itself; that is what the data checker and the manual checklist are for.
Full results, sample reports and the demo: https://m-shahmeer-kz.github.io/ml-pipeline-auditor/
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Code Quality & Review.
Static analysis reads Python code only: leaks that happen in SQL, in another repository or inside the data itself need the data split check and the manual checklist. Rules are tuned for scikit-learn, imbalanced-learn, pandas, XGBoost/LightGBM, Keras and PyTorch idioms. The data split check needs pandas and numpy.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 1 day ago
- Software
Python 3.9+
- Optional
- pandasnumpy (only for the data split check)
Runs fully on your machine. If your agent receives the skill as text, it saves the two scripts to .mlaudit/scripts/ in your project first, then runs them.
Permissions
File Scopes
Terminal: runs two local Python scripts (code analyser, data split check). Read files: your .py/.ipynb code, requirements.txt and optional train/test data. Write files: the audit report, scratch files in .mlaudit/ (deleted at the end) and, only with your OK, the code fixes. No network; nothing is uploaded.