Mlops Pipeline Generator

# MLOps Pipeline Generator Your model works in the notebook. Now make it work in prod — with retraining, monitoring, and rollbacks. This skill is the production playbook for the 80% of MLOps that takes 80% of the time. ## What it does Generates a complete MLOps pipeline: - **Training** — reproducible, parameterized, versioned data + code - **Evaluation** — accuracy, fairness, calibration, regression tests - **Registry** — MLflow model registry with stage transitions - **Deployment** — canary, blue/green, shadow modes - **Monitoring** — data drift, prediction drift, performance decay - **Retraining** — schedule-based, drift-triggered, performance-triggered - **Rollback** — automatic on performance regression - **Audit trail** — every prediction traceable to model + data version ## When to use it - You have a working model in a notebook and need to ship it - Your model is in prod but you can't reproduce training - Drift is killing your model performance and you don't know - You need to retrain regularly but it's a manual mess - Your team can't deploy ML models without a 2-week process - Compliance asks "how do you know the model in prod is the one you tested?" ## Why it's better than ad-hoc prompting Most "productionize my ML" prompts give toy examples. This skill is different: - **End-to-end** — from training to rollback, not just deployment - **Framework-agnostic** — scikit-learn, PyTorch, XGBoost, HuggingFace - **Real orchestration** — Airflow / Kubeflow / Prefect, not "just cron it" - **Drift detection done right** — KS-test, PSI, embedding distance - **Cost-aware** — spot instances, autoscaling, model quantization ## Architecture ``` ┌─────────────────────────────────────────────────────────┐ │ Agent (Claude/Cursor) │ │ - Reads your notebook / training script │ │ - Asks about data, framework, deployment target │ │ - Generates pipeline from template │ └───────────────┬─────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ skills/mlops-pipeline-generator/ │ │ scripts/ │ │ ├── gen_pipeline.py # Main pipeline generator │ │ ├── gen_dag.py # Airflow DAG │ │ ├── gen_evaluate.py # Eval + fairness tests │ │ ├── gen_drift.py # Drift detection │ │ ├── gen_retrain.py # Auto-retrain triggers │ │ ├── gen_deploy.py # Canary/blue-green │ │ ├── gen_rollback.py # Auto-rollback │ │ └── gen_audit.py # Audit trail + lineage │ │ references/ │ │ ├── architecture.md # End-to-end diagram │ │ ├── drift-guide.md # Data + prediction drift │ │ ├── deployment-modes.md # Canary vs blue-green │ │ ├── cost-optimization.md │ │ └── compliance-evidence.md │ │ templates/ │ │ ├── airflow-dag.py │ │ ├── mlflow-train.py │ │ ├── seldon-deploy.yaml │ │ ├── drift-monitor.py │ │ └── retrain-trigger.py │ └─────────────────────────────────────────────────────────┘ ``` ## Quick start ```bash # 1. Install pip install mlflow scikit-learn evidently pandas # 2. Generate the full pipeline python scripts/gen_pipeline.py --framework pytorch --data csv --target col_y --out pipeline/ # 3. Generate just the Airflow DAG python scripts/gen_dag.py --schedule "0 6 * * *" --tasks train,evaluate,deploy --out dag.py # 4. Generate drift detection python scripts/gen_drift.py --reference data/train.csv --features col1,col2,col3 --out drift.py # 5. Generate retrain trigger python scripts/gen_retrain.py --trigger drift --threshold 0.1 --out retrain.py # 6. Generate canary deployment python scripts/gen_deploy.py --mode canary --traffic-split 10/90 --out deploy.yaml ``` ## The 6 pipeline stages ``` ┌─────────────┐ ┌──────────────┐ ┌─────────────┐ │ 1. Extract │──▶│ 2. Validate │──▶│ 3. Train │ │ data │ │ schema │ │ model │ │ versioned │ │ quality │ │ versioned │ └─────────────┘ └──────────────┘ └─────────────┘ │ ▼ ┌─────────────┐ ┌──────────────┐ ┌─────────────┐ │ 6. Monitor │◀──│ 5. Deploy │◀──│ 4. Evaluate │ │ + retrain │ │ canary/blue │ │ + register │ │ triggers │ │ green │ │ in MLflow │ └─────────────┘ └──────────────┘ └─────────────┘ ``` ### Stage 1: Extract - Pull from data warehouse (Snowflake, BigQuery, Postgres) - Version the dataset (DVC, lakeFS, delta lake) - Snapshot to object storage with content hash ### Stage 2: Validate - Schema match (Great Expectations, Pandera) - Data quality (nulls, outliers, drift vs reference) - PII detection (if applicable) - BLOCK on quality failure ### Stage 3: Train - Parameterized (config + CLI args) - Reproducible (pinned dependencies, random seed, data version) - Distributed (Dask, Ray, Horovod for big models) - Track with MLflow (params, metrics, artifacts) ### Stage 4: Evaluate - Holdout accuracy + regression tests - Fairness across protected groups - Calibration (if classification) - Performance vs previous model (BLOCK if worse) - SHAP / feature importance ### Stage 5: Deploy - Canary (10% traffic) → shadow (no traffic) → blue-green - Health check (latency, error rate, prediction distribution) - Auto-rollback if performance drops - A/B test for business metrics ### Stage 6: Monitor + retrain - Data drift (KS-test, PSI, embedding distance) - Prediction drift (output distribution shift) - Performance decay (need labels — delayed feedback) - Auto-retrain triggers (drift, schedule, performance) - Audit trail (every prediction → model version → data version) ## Drift detection methods | Method | Best for | How it works | |--------|----------|--------------| | **KS test** | Continuous features | Compare distributions of new vs reference | | **PSI** | Categorical + continuous | Population Stability Index, threshold 0.1-0.25 | | **Chi-square** | Categorical | Compare category frequencies | | **Embedding distance** | Text / image | Cosine similarity of embeddings | | **Prediction drift** | Any | Output distribution shifts | | **Performance decay** | With labels | Accuracy drop over time window | ## Pricing Single-purchase, lifetime access. $18.00. Includes: - 7 Python pipeline scripts - 5 reference docs (architecture, drift, deployment, cost, compliance) - 5 production templates (Airflow, MLflow, Seldon, drift, retrain) - Framework support: scikit-learn, PyTorch, XGBoost, HuggingFace - Future updates for the same major version ## Example usage > "I have a scikit-learn model in a notebook. It's a fraud classifier. Generate the full MLOps pipeline." The skill will: 1. Extract the training code from the notebook 2. Generate parameterized version 3. Create Airflow DAG with 6 stages 4. Generate MLflow training script 5. Generate evaluation script (with fairness checks) 6. Generate Seldon deployment YAML (canary) 7. Generate drift monitoring (Evidently) 8. Generate retrain trigger (drift + performance) 9. Output `mlops-pipeline/` ready to deploy ## Compatibility Works with any agent that supports the SKILL.md standard and can execute Python: Claude Code, OpenClaw, Codex CLI, Cursor, Gemini CLI, Cline, Windsurf, Aider. Frameworks: scikit-learn, PyTorch, XGBoost, LightGBM, HuggingFace, TensorFlow. Orchestrators: Airflow, Kubeflow, Prefect, Dagster. Deployment: Seldon, KServe, BentoML, Ray Serve. Tested on Linux, macOS, Windows. ## Tags mlops, ml, machine-learning, pytorch, sklearn, mlflow, airflow, deployment, monitoring