Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+22 more

    ETL Pipeline Builder — Design Cleaner Data Transformation

    1

    Design explicit, testable ETL and ELT workflows with defined grain, safe reruns, data-quality checks, reconciliation and recovery behavior.

    $9.99

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    ETL Pipeline Builder — Design Cleaner Data Transformation

    ETL Pipeline Builder — Design Cleaner Data Transformation

    Example session with this skill installed

    Illustrative scenario: A daily CSV contains order_id and amount. order_id should be unique; amount must be a valid nonnegative decimal. No warehouse engine or incremental timestamp exists. Design a safe snapshot pipeline that quarantines invalid records and avoids silently choosing duplicate orders.

    • Read your context and instructions
    • Compiled the etl pipeline builder

    Contract: One row per order_id in each supplied daily snapshot. amount is a nonnegative decimal; allowed precision and currency must be confirmed. Snapshot completeness and deletion meaning remain unknown.

    1. Landing: Preserve the original authorized file with batch ID and checksum for diagnosis and replay. Reject unreadable files without publishing partial output.
    2. Staging: Parse both fields explicitly. Keep raw values and validation reasons. Treat missing IDs, invalid decimals and negative amounts as failures.
    3. Duplicate handling: Detect repeated order_id values. Quarantine all rows for the ambiguous key or stop the batch under the agreed policy; never select an arbitrary winner.
    4. Validation: Report landed, valid and quarantined counts. Reconcile landed = valid + quarantined under exclusive classification. Validate unique keys and nonnegative amounts in the accepted set. Compare totals only for comparable populations.
    5. Publication: Publish a versioned validated snapshot only after the agreed quality gate. Exact transactional replacement or pointer-swap behavior depends on the target engine. Retain the previous successful version for rollback.
    6. Reruns: Same file checksum must not create duplicate published snapshots. Use a controlled batch manifest and replay policy.
    7. Observability: Record source file, batch ID, counts, failures, status and target version. Alert on failures only through configured authorized tooling.

    Tests: Valid file, missing ID, duplicate ID, malformed/negative amount, empty file, failed publication and identical rerun. Verify source-to-target reconciliation and rollback on representative data.

    Unknowns: Currency/precision, snapshot completeness, deletion semantics, target transactions, storage and retention. No incremental cursor invented.

    Execution: Design only; no files loaded, tests run or deployment completed.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    ETL Pipeline Builder helps data and analytics engineers design or review source-to-target transformation workflows. It clarifies business meaning, grain, keys, schema contracts and consumer expectations before proposing extraction, staging, cleansing, transformation, validation and publication steps.

    The skill supports vendor-neutral SQL, Python, Spark, warehouse and orchestration designs, with platform-specific guidance only when context is supplied. It plans deterministic transformations, incremental boundaries, schema changes, late data, backfills, idempotency and source-to-target reconciliation. Correctness comes before performance optimization.

    Works in ChatGPT and Claude with supplied schemas and authorized data. Generated designs and code require representative testing in the exact environment. It does not invent schemas, row counts, connector behavior, performance gains or successful production deployments.

    How to use

    Skills on Agensi run inside your AI agent. Copy the prompt, paste it into your agent, and it connects to Agensi and installs this skill for you. If a payment is needed, your agent shows it to you first.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill passes our trust checks below.

    Security scanned

    Passed our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scanned

    Verified clean 7 days ago

    • Passed all security checks, Safe to install

    Listed7 days ago

    What's inside

    Frequently Asked Questions