dataforge

    2

    A manifest-driven toolkit for building, linting, and running Databricks and DuckDB data pipelines.

    Free

    1 installSecurity scanned

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+22 more

    dataforge

    Example session with this skill installed

    dataforge build a new metric for daily active users based on the logins table in the silver catalog, then open a PR.

    • Read your context and instructions
    • Compiled the dataforge

    I have processed the request for the daily_active_users metric. I scanned the silver.logins schema from the manifest, generated the transformation SQL, and validated the pipeline configuration.

    Metric Pipeline: Daily Active Users

    1. Transformation SQL (src/gold/m_daily_active_users.sql)

    -- Metric: Daily Active Users
-- Description: Count of unique user_ids per day from silver.logins
SELECT
    CAST(login_timestamp AS DATE) AS event_date,
    COUNT(DISTINCT user_id) AS dau_count
FROM
    silver.logins
GROUP BY
    1
ORDER BY
    1 DESC;

    2. Job Configuration (conf/jobs/dau_pipeline.yaml)

    name: dau_pipeline
tasks:
  - task_key: calculate_dau
    sql_task:
      query_path: src/gold/m_daily_active_users.sql
    warehouse_id: ${dataforge.warehouse_id}
schedule:
  quartz_cron_expression: "0 0 2 * * ?"
  timezone_id: UTC

    3. Verification Results

    • SQL Syntax: PASSED (DuckDB dialect check)
    • Schema Alignment: PASSED (References silver.logins.user_id)
    • Standards Check: PASSED (Naming convention m_* enforced)

    4. Pull Request Created

    • Branch: feature/metric-dau
    • PR URL: https://github.com/org/repo/pull/42
    • Status: Open - Awaiting CI/CD

    Next steps

    1. Review the generated SQL in src/gold/m_daily_active_users.sql.
    2. Run dataforge run job dau_pipeline to test the execution in your dev environment.
    3. Merge the pull request to deploy the schedule to production.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Data engineering teams often struggle with inconsistent pipeline standards, fragmented warehouse access, and the manual toil of turning requirements into production code. Dataforge provides a manifest-driven toolkit to automate the lifecycle of data pipelines from discovery to deployment. It enforces a unified interface for interacting with your warehouse while ensuring all code adheres to project-specific standards.

    What it does

    • Warehouse Discovery scans your codebase and warehouse to generate a central manifest of truths.
    • Unified Querying executes SQL, explores catalogs, and monitors jobs across Databricks and DuckDB through one interface.
    • Automated Pipeline Building converts GitHub issues or natural language requests into full metric pipelines and pull requests.
    • Standards Enforcement lints pipeline code, SQL schemas, and job configurations against manifest definitions.
    • Platform Monitoring triggers and tracks job execution status directly within the agent environment.

    How it works

    Four cooperating agent skills that build data pipelines with no knowledge of your data baked in. Everything business-specific (zones, catalogs, domains, grains, naming) lives in one generated manifest; the skills read it and never contain it.

    1. forge-init inspects your warehouse and codebase and writes the manifest from an eight-line seed.
    2. forge-dbx is one interface for queries, jobs, and catalogs on Databricks or DuckDB.
    3. forge-lint enforces naming, SQL formatting, join strategy, and safety gates before code is written.
    4. forge-builder takes a GitHub issue to a reviewed pull request in seven phases, with a human checkpoint at each.

    What it runs

    local CLIs only (databricks or duckdb, gh, git, python for a bundled verification script). It makes no network calls of its own and asks for confirmation at every point that changes anything.

    Frameworks & tools

    This skill utilizes the databricks CLI, duckdb, gh for GitHub integration, git, and python. It is designed for data platforms running on Databricks or local DuckDB instances. Try it with no cloud account: the bundled examples/support-desk is a local DuckDB project. Run python seed.py, then say "initialize dataforge".

    Why this beats prompting it yourself

    Generic prompts often hallucinate table schemas or ignore local project conventions. Dataforge uses a resolved manifest as a single source of truth, ensuring the agent always knows your specific catalog structure and pipeline standards.

    Use cases

    • Generate a new Databricks metric pipeline from a GitHub feature request.
    • Audit existing SQL scripts against project-wide schema and naming standards.
    • Explore DuckDB catalogs and run local data transformations without leaving the IDE.
    • Initialize a new data project by scaffolding a manifest from an existing warehouse.

    Known limitations

    Requires local installation of Databricks CLI or DuckDB. All project-specific data must be defined in the manifest to ensure accuracy.

    MIT licensed

    Source, issues, and releases https://github.com/tewarishishir/dataforge

    How to use

    Skills on Agensi run inside your AI agent. Copy the prompt, paste it into your agent, and it connects to Agensi and installs this skill for you. If a payment is needed, your agent shows it to you first.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill passes our trust checks below.

    Security scanned

    Passed our 8-point scan before listing

    1 install

    Downloaded by developers to date

    Free forever

    No account required to browse

    Trust & safety

    Security scanned

    Verified clean 11 days ago

    • Free to download with an account

    Requires

    Python
    macOS

    Listed11 days ago

    What's inside

    Frequently Asked Questions