- Home
- Skills
- APIs & Backend
- Background Job Queue Reliability for BullMQ Celery Sidekiq
Works with the AI Agents you already use
Background Job Queue Reliability for BullMQ Celery Sidekiq
Stop background jobs being lost, run twice or stuck: idempotency, retries, dead letters, timeouts, graceful deploys, crash-tested. RELIABLE / AT RISK / LOSING JOBS.
$5
Background Job Queue Reliability for BullMQ Celery Sidekiq
Example session with this skill installed
Audit our Celery setup. Django app, Redis broker, workers on Kubernetes. Tasks: send_invoice_email, charge_subscription, generate_report (up to 2 hours), sync_crm. Sometimes customers get two invoices and some reports never finish.
- Read your context and instructions
- Compiled the background job queue
Job queue reliability: billing — VERDICT: LOSING JOBS
Stack: Celery 5.5 on Redis, 4 pods × concurrency 8, terminationGracePeriodSeconds 30
Job inventory
| Job | Side effect | Max runtime | Retries | Timeout | Idempotent | After commit |
|---|---|---|---|---|---|---|
| send_invoice_email | 3 s | 3 | none | no | no | |
| charge_subscription | Stripe charge | 4 s | 3 | none | no | yes |
| generate_report | S3 file | 2 h | 0 | none | yes | yes |
| sync_crm | HubSpot API | 20 s | 5 | none | yes (upsert) | no |
Findings
| # | Severity | Job | Risk | Evidence | Fix |
|---|---|---|---|---|---|
| 1 | Critical | generate_report | Redis visibility timeout is 1 h (default); a 2 h task is redelivered and runs twice, and pods are killed after 30 s on deploy, so the report is lost | settings.py:212, no visibility_timeout | split into chunks with checkpoints; acks_late + task_reject_on_worker_lost; run on a dedicated queue |
| 2 | High | send_invoice_email | enqueued inside atomic(); plus no idempotency, so a redelivery sends a second invoice | billing/services.py:88 | delay_on_commit() and an InvoiceEmailSent unique row per invoice |
| 3 | High | charge_subscription | retry can double-charge | billing/tasks.py:41 | pass Stripe idempotency_key=f"charge-{sub.id}-{period}" |
| 4 | Medium | all | early ack: pod OOM kill loses in-flight tasks | default task_acks_late=False | enable for idempotent tasks only |
| 5 | Medium | all | no failed-task alert | none | export task failures and queue length to your metrics; alert on failures > 0 in 10 min |
Tests run
| Test | Result |
|---|---|
| SIGKILL during send_invoice_email (after fix) | redelivered, 1 email sent |
| Rolled-back transaction | no task ran |
| charge retry with same idempotency key | 1 Stripe charge |
| generate_report crash test | not run: needs staging data |
Verdict moves to AT RISK after fixes 1 to 3, and to RELIABLE once the report crash test passes in staging.
Connects securely to your tools. The creator never sees your data.
About this skill
Background queues fail quietly. A Celery task is acknowledged before it runs and vanishes when the worker is killed. A BullMQ job runs longer than its lock and a second worker starts it again. A Sidekiq job is enqueued inside a transaction that rolls back. A deploy kills workers mid-job. Customers get two emails, two charges, or nothing at all.
This skill gives your coding agent an audit procedure and a reference file of library defaults checked against the official docs. It:
- Inventories every job: side effects, runtime, retries, timeouts, who enqueues it, and how workers stop on deploy
- Checks 12 risks per job: idempotency, enqueue-after-commit, retries with backoff and a cap, dead-letter path, timeouts vs runtime, graceful shutdown, ack timing, payload size, duplicate schedulers, concurrency vs DB pool and API limits, retention, and monitoring
- Knows the traps: Celery's early ack and the Redis visibility timeout with long ETAs, BullMQ's 30-second lock and
close()with no timeout, Sidekiq'sretry: falseskipping the Dead set, Laravel'sretry_aftervs--timeout, SQS visibility and redrive - Fixes data-loss risks first, then duplicates, then operations, with small diffs
- Proves the fixes with a crash test (SIGKILL mid-job), a graceful-stop test, retry and dead-letter tests, and a rolled-back transaction test
- Writes a short runbook: replay dead jobs, pause a queue, what to alert on
- Returns a verdict: RELIABLE, AT RISK or LOSING JOBS
Works with: BullMQ, Celery, Sidekiq, RQ, Dramatiq, Oban, Laravel queues, Solid Queue, GoodJob, SQS and RabbitMQ.
Safety: never purges, deletes or replays jobs in a real queue, and asks before changing production retry, timeout or concurrency settings.
Honesty note: it quotes defaults from the reference file and asks you to confirm them against your installed version, and only reports tests it ran.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for APIs & Backend.
Cannot replay or purge jobs in live queues; you run the provided commands. Library defaults must be confirmed against your installed version. Queues not listed get the generic 12-point checklist without library-specific defaults. Crash tests need a local or staging environment.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 29 days ago
- Software
- Queue libraryQueue broker
Use your own broker, for example Redis, RabbitMQ, SQS or PostgreSQL.