- Home
- Skills
- Agents & Orchestration
- ML Notebook Audit & Repair
More screenshots
Works with the AI Agents you already use
ML Notebook Audit & Repair
Finds and fixes data leakage, test-set misuse and unsupported conclusions in ML notebooks, then re-audits until clean.
$25
ML Notebook Audit & Repair
Example session with this skill installed
Audit breast_cancer_classifier.ipynb with the ML notebook audit skill. Fix anything severe and tell me if I can trust the conclusion.
- Read your context and instructions
- Compiled the ml notebook audit
Summary
The notebook claimed "99% accuracy on unseen data". The audit found 3 severe defects, patched them, executed the notebook end to end and re-audited it.
Status: PASS. The honest result is 97.4%, and the "clearly better" model is not clearly better.
Severe findings (all patched)
| ID | Sev. | Location | Problem | Fix |
|---|---|---|---|---|
| F-001 | 9 | cell 5 | StandardScaler fit on all data before the split: test statistics leak into training | Split first; scaler inside a pipeline |
| F-002 | 10 | cells 7, 10 | C tuned on the test set, then the same test set reports the headline metric | 5-fold CV on train only; test used once |
| F-003 | 9 | markdown | Claims "99% on unseen data", "balanced" (it is 37/63), "clearly beats", "fully reproducible" (no seeds) | Seeds, balanced metrics, bootstrap CI, rewritten conclusion |
The original gave 0.947, 0.982 and 0.982 across three runs, and the "best" C changed between runs.
Corrected results (held-out test, 114 samples)
| Model | Accuracy | Balanced acc. | F1 malignant |
|---|---|---|---|
| Logistic regression | 0.974 | 0.969 | 0.964 |
| Random forest | 0.956 | 0.950 | 0.940 |
Bootstrap 95% CI for the difference: [-0.027, 0.067]. It includes 0, so there is no clear winner.
| Random forest | 0.956 | 0.950 | 0.940 |
Bootstrap 95% CI for the difference: [-0.027, 0.067]. It includes 0, so there is no clear winner.
lr_search = GridSearchCV(
make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)),
{"logisticregression__C": [0.001, 0.01, 0.1, 1, 10, 100]},
cv=StratifiedKFold(5, shuffle=True, random_state=SEED),
scoring="balanced_accuracy",
).fit(X_train, y_train)
Final status
Unresolved findings > 8: 0 · Evidence level 3 (fully executed, reproducible across runs) · PASS
Next steps
- Report 0.969 balanced accuracy, not 99%.
- Present logistic regression as "simpler and at least as good", not "better".
- Rerun with other seeds if you need a stronger claim.
Connects securely to your tools. The creator never sees your data.
About this skill
Most ML notebooks run fine and are still wrong. The scaler was fit before the split, the hyperparameters were tuned on the test set, the plot says "validation" but shows test data, and the conclusion claims more than the numbers support.
This skill makes your agent audit the notebook as one experiment, not cell by cell.
How it works
- Traces data → split → preprocessing → training → model selection → evaluation → metrics → plots → conclusions.
- Rates every finding from 1 to 10 and patches only the severe ones (above 8), with the smallest correct change.
- Propagates each fix to metrics, plots, tables, artifacts, markdown and conclusions.
- Re-audits the whole notebook after every patch and checks for regressions.
- Ends with an honest PASS or FAIL report that states what was actually executed.
What it catches
- Data leakage and overlapping train/validation/test sets
- Model selection on the test set
- Misleading metrics on imbalanced data
- Training, checkpoint and train/eval-mode bugs
- Unfair comparisons between frozen, fine-tuned, PEFT/LoRA and from-scratch models
- Missing seeds and non-reproducible splits
- Markdown and conclusions that don't match the results
What's included
Full audit checklist, finding schema, patch protocol, report template, and two worked examples (scikit-learn and ResNet18).
Works best for PyTorch and scikit-learn notebooks.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Agents & Orchestration.
Automatic patches only apply to findings with severity > 8. Requires Level 4 evidence for full experimental validation. Static analysis only if datasets are missing.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 24 days ago
- Software
Python 3jupyter
Permissions
Reads the notebook, writes the repaired version, and runs Python/Jupyter in the terminal to execute it. No network access, no accounts, no credentials.