Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Event Driven Kafka Producer Consumer

    1

    Build reliable Kafka pipelines in Node.js and Python with idempotent producers, strict ordering, and DLQ support.

    $15

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Event Driven Kafka Producer Consumer

    Event Driven Kafka Producer Consumer

    Example session with this skill installed

    Generate a reliable Node.js Kafka producer and consumer for a payment processing system. I need strict message ordering by paymentId and a way to handle failed messages without losing them.

    • Read your context and instructions
    • Compiled the event driven kafka

    I have architected a robust Node.js Kafka pipeline using KafkaJS. The producer is configured for idempotency to prevent duplicate payments, and the consumer implements a manual commit strategy with a Dead Letter Queue (DLQ) for unrecoverable failures.

    Kafka Payment Processing Implementation

    // producer.js
    const { Kafka } = require('kafkajs');
    
    const kafka = new Kafka({
      clientId: 'payment-service',
      brokers: [process.env.KAFKA_BROKER]
    });
    
    const producer = kafka.producer({
      allowAutoTopicCreation: false,
      transactionalId: 'payment-producer-v1',
      maxInFlightRequests: 1,
      idempotent: true // enable.idempotence=true
    });
    
    async function sendPaymentEvent(payment) {
      await producer.connect();
      await producer.send({
        topic: 'payment-events',
        acks: -1, // acks=all
        messages: [
          { 
            key: payment.paymentId, // Partitioning by entity key for ordering
            value: JSON.stringify(payment) 
          }
        ],
      });
    }
    
    // consumer.js
    const consumer = kafka.consumer({ groupId: 'payment-group' });
    
    async function startConsumer() {
      await consumer.connect();
      await consumer.subscribe({ topic: 'payment-events', fromBeginning: true });
    
      await consumer.run({
        autoCommit: false, // Manual commit management
        eachMessage: async ({ topic, partition, message }) => {
          try {
            const paymentData = JSON.parse(message.value.toString());
            await processPayment(paymentData);
            
            // Commit only after successful processing
            await consumer.commitOffsets([
              { topic, partition, offset: (BigInt(message.offset) + 1n).toString() }
            ]);
          } catch (error) {
            console.error('Processing failed, routing to DLQ', error);
            await producer.send({
              topic: 'payment-events-dlq',
              messages: [{ key: message.key, value: message.value }]
            });
          }
        },
      });
    }
    

    Next steps

    1. Add the KAFKA_BROKER address to your .env file based on the .env.example provided.
    2. Verify that the payment-events-dlq topic is created in your Kafka cluster.
    3. Run the automated evaluation suite to test the consumer group rebalancing behavior.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Distributed messaging often fails due to lost messages, out-of-order processing, or poison pills that crash consumers. This skill provides production-ready Apache Kafka producer and consumer implementations in Node.js or Python, enforcing reliability through idempotent delivery and dead-letter queues.

    What it does

    • Idempotent Producers ensures messages are delivered exactly once even during retries by using acks=all.
    • Key-Based Partitioning maintains strict message ordering for specific entities like userId or orderId.
    • Manual Commit Management prevents data loss by acknowledging offsets only after successful message processing.
    • Dead Letter Queues (DLQ) isolates unrecoverable messages to prevent pipeline blockage and enable debugging.

    How it works

    1. Initialize Producer with enable.idempotence=true and automated retry logic for high-availability writes.
    2. Define Routing by business entity keys to ensure sequential processing within specific partitions.
    3. Configure Consumer Groups for horizontal scaling and fault-tolerant offset tracking.
    4. Implement Error Handling that automatically routes failed payloads to a designated DLQ topic.

    Frameworks & tools

    Uses KafkaJS for Node.js environments or Confluent-Kafka for Python. It follows standard patterns for distributed systems messaging and pub/sub architectures.

    Why this beats prompting it yourself

    Writing reliable Kafka code requires deep knowledge of configuration flags that prevent silent failures. This skill enforces strict ordering and fault tolerance by default, saving you from troubleshooting partition skews or data duplication in production.

    Use cases

    • Build resilient order processing pipelines for e-commerce backends.
    • Implement real-time user activity tracking with strict sequential requirements.
    • Create fault-tolerant data ingestion layers for analytics platforms.
    • Manage unrecoverable message failures without stopping the entire stream.

    Known limitations

    Focuses on application-level logic for KafkaJS and Python. It does not handle infrastructure provisioning or Kafka cluster management.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for APIs & Backend.

    Focuses on KafkaJS (Node.js) and Python (Confluent-Kafka). Does not perform infrastructure setup, cluster provisioning, or global system package installation.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 3 days ago

    Permissions

    Read Files

    Target Use Cases: - Build an event-driven Kafka producer and consumer in Node.js. - Configure Kafka consumer group with Dead Letter Queue handling. - Implement idempotent message publishing in Apache Kafka.

    Listed3 days ago

    What's inside

    Frequently Asked Questions

    This skill provides production-ready Kafka implementation patterns for Node.js (KafkaJS) and Python (confluent-kafka), focusing on high-reliability features like idempotent delivery and Dead Letter Queues.

    This skill solves the 'poison pill' problem and data duplication by enforcing idempotent producers and manual offset commits, ensuring messages are processed exactly once and in the correct order.

    You will receive optimized configuration templates and architectural patterns for both producers and consumers, including pre-built logic for DLQ routing and key-based partitioning.

    This skill is designed for application-level logic; you will need an existing Kafka cluster (like Confluent Cloud, Aiven, or a local Docker instance) to connect to.

    Yes, the implementations are built on standard Kafka protocols, making them compatible with any AI agent or backend service capable of executing Node.js or Python code.

    While it handles complex configurations like 'acks=all' and 'enable.idempotence' by default, the skill is modular, allowing you to adjust retry limits and batching settings to fit your specific latency requirements.

    New in APIs & Backend