- Home
- Skills
- Data & Databases
- Data Pipeline Failure Diagnosis Recovery Architect
Works with the AI Agents you already use
Data Pipeline Failure Diagnosis Recovery Architect
Diagnose data pipeline failures and design safe, idempotent recovery plans to restore reliable data delivery.
$9.99
Data Pipeline Failure Diagnosis Recovery Architect
Example session with this skill installed
I have a failed Airflow task in a dbt pipeline. The error says 'Unique constraint violation' on the sales_summary table. It started yesterday at 14:00 UTC. What should I do?
- Read your context and instructions
- Compiled the data pipeline failure
I have analyzed the unique constraint violation in your Airflow dbt pipeline. The failure suggests an idempotency issue where a partial write from a previous attempt or an overlap in backfill windows is causing duplicate keys in the sales_summary table.
Pipeline Incident Diagnosis & Recovery Plan
| Component | Assessment |
|---|---|
| Incident Scope | sales_summary table, starting 2023-10-26 14:00 UTC |
| Failure Boundary | Transformation/Load (dbt write to destination) |
| Risk Level | High (Potential duplicate data in downstream reporting) |
| Confidence | 85% (Common symptom of non-atomic incremental loads) |
Recovery Runbook
Evidence Preservation: Query the sales_summary table for the affected timestamp and identify the specific duplicate keys and their inserted_at metadata.
2.
Safety Check: Verify if the dbt model uses incremental logic and if a unique_key is defined in the configuration.
3. Isolation: Temporarily pause the Airflow DAG to prevent repeated failures and log bloat.
4. Execution:
- If using Snowflake/BigQuery: Run a manual DELETE on the affected partition or records matching the duplicate keys.
- If using dbt: Execute a targeted dbt run --select sales_summary --full-refresh if the table size allows, or a manual merge to deduplicate.
5.
Validation: Run SELECT count(*), count(distinct primary_key) FROM sales_summary WHERE ... to ensure the constraint is met.
Next steps
- Provide the dbt model configuration for
sales_summaryto verify the incremental strategy. - Check if any upstream source system started producing duplicate records at 14:00 UTC.
- Implement a
dbt-expectationstest for uniqueness to catch this before the load step in the future.
Connects securely to your tools. The creator never sees your data.
About this skill
Data pipeline failures cause silent corruption, downstream reporting errors, and expensive manual cleanup. This skill provides a systematic framework to diagnose root causes across the entire data lifecycle and design safe, idempotent recovery plans.
What it does
- Incident triage determines the scope of data loss, affected time windows, and downstream business impact.
- Root cause diagnosis traces failures across ingestion, transformation, orchestration, and storage layers.
- Recovery strategy design selects the safest method among retries, resumes, replays, backfills, or restores.
- Integrity validation defines specific checks for row counts, schema drift, and business invariants before data release.
- Prevention planning generates post-incident reports and reliability roadmaps to close monitoring gaps.
How it works
- Intake triggers an assessment of the platform, architecture, error logs, and recent deployment history.
- Boundary identification locates the earliest point where data behavior diverged from expectations.
- Safety assessment evaluates idempotency, partial commit risks, and downstream side effects.
- Runbook generation provides ordered execution steps with stop conditions and rollback procedures.
Frameworks & tools
Supports SQL, Python, Spark, and dbt. Works with orchestration tools like Airflow and cloud data platforms. Applicable to both batch and streaming architectures involving message queues and object storage.
Why this beats prompting it yourself
Generic prompts often suggest destructive actions like clearing checkpoints or broad retries without checking for side effects. This skill enforces a safety-first methodology that preserves evidence and ensures recovery is reversible and validated.
Use cases
- Diagnosing a stalled streaming pipeline with growing consumer lag and checkpoint mismatch.
- Recovering from a transformation defect that corrupted historical partitions in a data warehouse.
- Troubleshooting authentication failures or rate limits during high-volume source extraction.
- Designing a safe backfill for a new data contract after schema drift occurred upstream.
Known limitations
Does not have direct system access and requires manual log/metric input. Cannot generate platform-specific commands without version details. Redaction of secrets must be handled by the user.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Data & Databases.
Requires manual log/metric input. Cannot generate platform-specific commands without version details. User must redact credentials. No direct system execution.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
1 use
Times used to date
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 8 days ago
Permissions
No special permissions declared or detected