More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Streaming and Processing at Scale

    1

    Process data with Spark, Kafka, Flink, ClickHouse, DuckDB and Polars. 10 ordered skills, starting with Apache Spark, Apache Kafka and Streaming, Apache Flink.

    Free

    Security scanned
    Streaming and Processing at Scale

    Streaming and Processing at Scale

    Example session with this skill installed

    My Spark job is failing with OutOfMemory errors on the executors during a join. I have one large table and one 500MB dimension table. How do I fix this?

    • Read your context and instructions
    • Compiled the streaming and processing

    The executor OOM during a join usually indicates a shuffle spill or an unsuccessful broadcast attempt. Given your dimension table is 500MB, it exceeds the default spark.sql.autoBroadcastJoinThreshold of 10MB.

    Spark Join Optimization Plan

    Action ItemConfiguration / Code ChangeReason
    Increase Broadcast Thresholdspark.conf.set("spark.sql.autoBroadcastJoinThreshold", "600mb")Forces the 500MB table to be broadcast to all executors, avoiding a costly shuffle.
    Explicit Broadcast Hintdf1.join(broadcast(df_dim), "key")Ensures Spark attempts a BroadcastHashJoin even if metadata is missing.
    Check for Data Skewdf1.select("key").groupBy("key").count().sort(desc("count"))Identifies if a single key is too large for one executor's memory.
    Increase Parallelismspark.conf.set("spark.sql.shuffle.partitions", "auto")Uses Adaptive Query Execution (AQE) to balance partition sizes.

    Diagnostic Verification

    To confirm the fix, check the Spark UI SQL Tab:

    1. Locate the failed Join node in the execution graph.
    2. Verify if it shows SortMergeJoin (bad for this case) instead of BroadcastHashJoin.
    3. Check Peak Execution Memory to see if the spill to disk exceeded available RAM.

    Next steps

    • Apply the broadcast() hint in your PySpark code and re-run.
    • Enable Adaptive Query Execution with spark.sql.adaptive.enabled=true.
    • Monitor the Spark UI for "Shuffle Read Size" to ensure the broadcast is working.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    The problem

    Advanced data engineers often need a repeatable process, not a one-off prompt. This pack targets: Process data with Spark, Kafka, Flink, ClickHouse, DuckDB and Polars.

    What's inside (10 skills, suggested learning order)

    1. Apache Spark — Distributed processing.
    2. Apache Kafka and Streaming — Real-time data.
    3. Apache Flink — Stateful stream processing.
    4. Real-Time Streaming Analytics — Real-time dashboards.
    5. ClickHouse — High-performance OLAP.
    6. DuckDB and Local Analytics — Fast analytics on files.
    7. Polars — High-performance DataFrames.
    8. IoT Data at Scale — Telemetry ingestion and analysis.
    9. Geospatial Data — Analyze location data.
    10. Web Scraping and Data Collection — Extract web data legally.

    Why a skill pack instead of a one-off prompt

    Each skill gives the agent a defined trigger, step-by-step procedure and output format, so results are consistent across sessions and easier to review.

    How to use

    1. Install the pack in your SKILL.md-compatible agent.
    2. Ask for the task in plain language (for example: "distributed processing").
    3. Review the output against the skill's quality checks before using it.

    Known limitations

    • Outputs are drafts that need human review; they are not professional, legal or financial advice.
    • Tools, APIs and prices change; check current vendor documentation before relying on version-specific details.
    • Quality depends on the context you give the agent (data, requirements, constraints).
    • Tested only where stated in the Compatibility note.

    Also by Rodrigo Santos

    Data Annotation and Labeling; Cloud Data Platforms and Data for AI; Customer and Sales Analytics

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Data & Databases.

    Focuses on data engineering and analytics. Does not cover hardware, firmware, or satellite imagery. Not for bypassing security controls or legal advice.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free to use

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 4 days ago

    • Free with an Agensi account

    Nothing to install. It runs in your AI agent.

    Instruction-only skills. No network access, shell commands or file writes are required; each skill reads the context you provide and returns a document.

    Permissions

    Read Files
    Listed4 days ago

    What's inside

    Frequently Asked Questions

    This skill is designed for any developer or data engineer using AI agents to build or fix data stacks involving Spark, Flink, Kafka, and OLAP databases like ClickHouse. It bridges the gap between generic coding and specialized distributed systems engineering.

    The package includes specialized reasoning modules for stream processing, big data tuning, and analytical storage optimization. You also get specific diagnostic checklists to use with Spark UI and Kafka metrics to identify bottlenecks.

    Unlike generic AI prompts, this skill uses a diagnosis-first methodology that requires evidence like logs or schemas before suggesting changes. This prevents the "hallucinated configurations" that often lead to further out-of-memory errors in distributed environments.

    The skill supports the most popular frameworks in the modern data stack, including Apache Spark, Flink, and Kafka, as well as high-performance local tools like Polars and DuckDB. It also handles geospatial data via PostGIS.

    Simply import the skill into your compatible AI agent platform; it is designed to work with most major LLM-based assistants that support external skill or tool integration. No complex local installation is required beyond providing your agent with the skill definitions.

    While it is excellent at designing and tuning data pipelines, it does not cover physical hardware selection, IoT firmware, or legal advice regarding web scraping. It focuses strictly on the software and data architecture layers.

    New in Data & Databases

    Free