Repair Lost Events Across Services

The problem

Event-driven systems often suffer from "silent" data loss where a message disappears between the producer and the database. Developers struggle to distinguish between network delays, consumer lag, misconfigured retention, or early acknowledgments that kill redelivery.

What it does

• Traces logical event identities across producers, outboxes, brokers, and consumers to find the exact loss boundary.
• Identifies failures in the delivery chain, such as commit-before-persist bugs or consumer checkpoint skipping.
• Detects "poison events" trapped in dead-letter queues or suppressed by incorrect deduplication logic.
• Implements architecture-compatible repairs like transactional outboxes, stable correlation IDs, and idempotent replays.
• Validates event paths using deterministic failure injection and end-to-end verification tests.

Frameworks & tools

Works with message brokers and streaming platforms including Kafka, RabbitMQ, SQS, Pub/Sub, and NATS. Supports various persistence layers and outbox patterns.

Why this beats prompting it yourself

Generic prompts often suggest "just checking logs" or blind replays that cause duplicate side effects. This skill enforces a strict evidence chain, requiring proof of persistence at every durable boundary before suggesting a fix.

Use cases

• Finding why a successful checkout event never triggered a fulfillment record.
• Diagnosing messages that vanish when a consumer service restarts during a rebalance.
• Repairing gaps in event streams caused by broker retention expiry or TTL settings.
• Auditing complex multi-hop event paths to ensure stable correlation and causation.

Known limitations

Requires a safe development or test environment for failure injection. Cannot recover historical events that were never durably recorded by the producer or broker.