More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Data Pipelines and Orchestration

    1

    Build ETL/ELT with dbt, Airflow, Dagster, CDC and reverse ETL. 10 ordered skills, starting with Data Engineering, ETL/ELT Pipelines, dbt (Data Build Tool).

    Free

    Security scanned
    Data Pipelines and Orchestration

    Data Pipelines and Orchestration

    Example session with this skill installed

    Review my Airflow DAG for performance issues and suggest a testing plan to ensure the data load is idempotent.

    • Read your context and instructions
    • Compiled the data pipelines

    I have reviewed the DAG and identified two critical performance bottlenecks: top-level database calls during parsing and a lack of idempotency in the task logic.

    Airflow DAG Review and Test Plan

    IssueSeverityRecommendation
    Top-level Variable.get()HighReplace with {{ var.value.key }} Jinja template to prevent scheduler lag.
    Blind INSERT logicHighReplace with MERGE or UPSERT using the data_interval_start as a partition key.
    Heavy importsMedMove import pandas inside the Python callable to speed up DAG parsing.

    Idempotency Test Plan

    import pytest
    from airflow.models import DagBag
    
    def test_dag_loading():
        dagbag = DagBag(dag_folder="dags/", include_examples=False)
        dag = dagbag.get_dag(dag_id="my_data_pipeline")
        assert dagbag.import_errors == {}
        assert dag is not None
    
    def test_idempotent_load_logic(mock_db_context):
        # Simulate running the same task twice on the same data interval
        task_input = {"start": "2023-01-01", "end": "2023-01-02"}
        run_1 = execute_load_task(task_input)
        run_2 = execute_load_task(task_input)
        
        # Verify no duplicate rows exist in target
        assert run_1.row_count == run_2.row_count
        assert get_target_count(task_input) == run_1.row_count
    

    Next steps

    1. Refactor the SQL operator to use a DELETE and INSERT pattern based on the execution date.
    2. Move all network-heavy logic into custom Airflow Hooks.
    3. Implement dbt tests on the target table to catch primary key violations after the Airflow run.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    The problem

    Data engineers often need a repeatable process, not a one-off prompt. This pack targets: Build ETL/ELT with dbt, Airflow, Dagster, CDC and reverse ETL.

    What's inside (10 skills, suggested learning order)

    1. Data Engineering — Design and maintain data pipelines.
    2. ETL/ELT Pipelines — Move and transform data.
    3. dbt (Data Build Tool) — Versioned analytical transformation.
    4. Apache Airflow — Orchestrate pipelines.
    5. Data Integration via APIs — Connect sources through APIs.
    6. Change Data Capture (CDC) — Replicate changes in real time.
    7. Reverse ETL — Activate warehouse data in tools.
    8. Python for Data Engineering — Scripts and pipelines in Python.
    9. Orchestration with Dagster and Prefect — Modern pipelines with built-in observability.
    10. Automated Data Pipeline Testing — Ensure quality on every run.

    Why a skill pack instead of a one-off prompt

    Each skill gives the agent a defined trigger, step-by-step procedure and output format, so results are consistent across sessions and easier to review.

    How to use

    1. Install the pack in your SKILL.md-compatible agent.
    2. Ask for the task in plain language (for example: "design and maintain data pipelines").
    3. Review the output against the skill's quality checks before using it.

    Known limitations

    • Outputs are drafts that need human review; they are not professional, legal or financial advice.
    • Tools, APIs and prices change; check current vendor documentation before relying on version-specific details.
    • Quality depends on the context you give the agent (data, requirements, constraints).
    • Tested only where stated in the Compatibility note.

    Also by Rodrigo Santos

    Data Annotation and Labeling; Cloud Data Platforms and Data for AI; Customer and Sales Analytics

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Data & Databases.

    Requires access to database logs for CDC. Does not provide hardware-specific infrastructure cost estimations or sizing.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free to use

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 4 days ago

    • Free with an Agensi account

    Software
    Python

    Instruction-only skills. No network access, shell commands or file writes are required; each skill reads the context you provide and returns a document.

    Permissions

    Read Files
    Listed4 days ago

    What's inside

    Frequently Asked Questions

    The skill includes pre-configured logic for Airflow DAG structures, dbt modeling patterns, CDC configuration templates, and testing suites for data quality validation. It provides the architectural 'brains' to build idempotent, failure-resistant pipelines without manual prompt engineering.

    This skill is compatible with any AI agent capable of code generation and architectural planning, specifically those optimized for Python and SQL development. It integrates seamlessly into workflows using Airflow, dbt Cloud/Core, and modern orchestrators like Dagster or Prefect.

    While 'Untitled Skill' specializes in Python-based tools like Polars and pandas, its dbt and SQL patterns are compatible with all major warehouses including Snowflake, BigQuery, and Redshift. For CDC, it specifically supports PostgreSQL and Debezium-compatible log-based extraction.

    It eliminates common failures like 'silent' data corruption and scheduler crashes by enforcing production-grade patterns like light top-level Airflow code and partition-aware incremental loading. It transforms generic AI code into reliable, memory-efficient data engineering logic.

    Yes, the skill includes specialized patterns for Reverse ETL orchestration. It provides logic to sync warehouse data to tools like Salesforce or HubSpot while specifically protecting manual human edits from being overwritten during the sync process.

    New in Data & Databases

    Free