Data Pipeline Quality Monitor

Automated Data Pipeline Reliability

The Datenpipeline-Qualitätsmonitor is a specialized diagnostic skill for developers and data engineers managing ETL processes, batch jobs, and data ingestion. It solves the "hidden failure" problem where pipelines run without crashing but produce degraded or incorrect data.

What it does

• Anomaly Detection: Analyzes throughput, error rates, and latency using Z-Score, seasonality, and drift detection to find outliers.
• Automated Context Harvesting: When a problem is detected, it automatically pulls associated logs, recent Git commits, and system metrics (CPU/RAM/Disk).
• Root Cause Analysis: Compares data drifts against code changes to determine if the issue is infrastructure-based, a code regression, or a change in the source data format.
• Actionable Recovery: Generates prioritized fixes, such as specific git reverts, system restarts, or targeted data re-runs.

Why use this skill?

Unlike generic monitoring tools that only alert you that a threshold was hit, this skill acts as a first responder. It performs the initial investigation by correlating 14% error spikes with specific regex changes in a PR from two hours ago. It saves hours of manual log diving and provides a structured JSON or visual report ready for stakeholders.

Output format

The skill produces comprehensive reports featuring metric tables (Current vs. Baseline), Z-Score severity ratings, log pattern summaries, and a prioritized checklist of correction measures (P0 to P1).