Queue-Based Architecture in Practice: BullMQ, RabbitMQ, SQS, Workers, Retries, DLQs, and Job Deduplication

Introduction

Every growing app eventually hits the same wall: some tasks (notifying thousands of people the moment someone likes a viral post, charging a card during a flash-sale checkout, delivering a payment confirmation across a banking network) take too long to do while a user is sitting there waiting. Queue-based architecture solves this by putting those tasks in a line and handling them in the background, one worker at a time. This guide explains queues, workers, retries, dead letter queues (DLQs), and job deduplication in plain language, using BullMQ, RabbitMQ, and Amazon SQS as real, working examples.

Quick Answer

A queue-based architecture separates "accepting a task" from "doing a task." A producer drops a task (a "job" or "message") onto a queue; a worker picks it up and processes it independently, at its own pace. If the worker fails, the system can retry automatically; if it keeps failing, the task lands in a dead letter queue for a human to inspect; and deduplication stops the same task from accidentally being processed twice. BullMQ, RabbitMQ, and SQS all implement this pattern, but with different trade-offs in setup, retry behavior, and hosting.

What Is a Queue-Based Architecture?

Think of a busy restaurant kitchen. A waiter doesn't cook your dish the moment you order; they write a ticket and clip it to a rail. Chefs (workers) pull tickets off the rail one at a time and cook them, at the kitchen's own pace, independent of how fast orders come in.

That ticket rail is the queue. In software:

• The producer is the waiter: it creates the task and adds it to the queue.
• The queue is the ticket rail: an ordered, durable holding area for tasks.
• The worker is the chef: a separate process that picks up tasks and does the actual work.

This decoupling is the whole point: the waiter (your web server) can keep taking orders even if the kitchen is backed up, instead of everyone waiting at the counter until one dish is done.

The Problem

Imagine a website that emails a receipt, updates inventory, and notifies a shipping partner every time someone places an order, all inside the same request that shows the user "Order confirmed!" If the email service is slow, or the shipping partner's API is briefly down, the customer sits there watching a spinner, or the whole request fails even though the order itself was fine.

Doing everything synchronously (immediately, in order, in the same request) means one slow or flaky step can block or break everything else. It also means a temporary failure, such as a timeout or a dropped connection, has nowhere to go except back to the user as an error.

The Solution

A queue-based architecture moves the slow or unreliable steps out of the request path and into the background:

Customer places order
        ↓
Web Server (producer)
        ↓
   Queue (BullMQ / RabbitMQ / SQS)
        ↓
   Worker picks up the job
        ↓
Success → done       Failure → retry → (still failing) → Dead Letter Queue

The customer gets an instant "Order confirmed," and the actual card charge, email, and notifications happen right after, off to the side. Automatic retries handle it if something hiccups, and the DLQ acts as a safety net if it never recovers.

Prerequisites

To follow along, you'll need:

• Node.js 18+ and npm
• A Redis instance (for BullMQ) and a RabbitMQ server
• An AWS account with an IAM user/role scoped to the queues you create for these examples, with sqs:CreateQueue, sqs:SendMessage, sqs:ReceiveMessage, sqs:DeleteMessage, sqs:GetQueueUrl, and sqs:GetQueueAttributes — not a blanket sqs:*
• Basic JavaScript knowledge (async/await)
• These packages: bullmq + ioredis (BullMQ), amqplib (RabbitMQ), @aws-sdk/client-sqs (SQS)

Meet the Tools: BullMQ vs RabbitMQ vs SQS

Restaurant analogy: BullMQ is like a ticket rail you build yourself out of parts you already have (Redis). RabbitMQ is like hiring a dedicated ticket-management company with its own staff and rules. SQS is like renting a ticket rail as a utility: you never see the rail itself, you just use it and pay for what you use.

Step-by-Step Implementation

1. BullMQ (Node.js + Redis)

Install and set up a queue plus a worker:

npm install bullmq ioredis

import { Queue, Worker } from 'bullmq';
const connection = { host: 'localhost', port: 6379 };

const ordersQueue = new Queue('orders', { connection });

const worker = new Worker(
  'orders',
  async (job) => {
    if (job.data.failTimes && job.attemptsMade < job.data.failTimes) {
      throw new Error('Simulated payment gateway timeout');
    }
    return { charged: job.data.amount, orderId: job.data.orderId };
  },
  { connection }
);

Retries with backoff

BullMQ has this built in via the attempts and backoff options:

await ordersQueue.add(
  'charge-card',
  { orderId: 'order-2', amount: 19.99, failTimes: 2 },
  { attempts: 3, backoff: { type: 'exponential', delay: 500 } }
);

This produces logs like the following. Order-2 fails twice, then succeeds on the 3rd try, and the delay between attempts roughly doubles (500ms, then about 1000ms), exactly as exponential backoff promises:

[05:10:01.833] WORKER picked up job 2 (charge-card) - attempt #1
[05:10:01.837] ❌ FAILED job 2 attempt 1/3: Simulated payment gateway timeout (attempt 1)
[05:10:02.384] WORKER picked up job 2 (charge-card) - attempt #2
[05:10:02.386] ❌ FAILED job 2 attempt 2/3: Simulated payment gateway timeout (attempt 2)
[05:10:03.403] WORKER picked up job 2 (charge-card) - attempt #3
[05:10:03.407] ✅ COMPLETED job 2 after 3 attempt(s): { charged: 19.99, orderId: 'order-2' }

Dead letter queue

BullMQ has no built-in DLQ, so you build one by listening for the failed event and checking if all attempts are used up:

const dlqQueue = new Queue('orders-dlq', { connection });

worker.on('failed', async (job, err) => {
  if (job.attemptsMade >= job.opts.attempts) {
    await dlqQueue.add('dead-order', { originalId: job.id, data: job.data, failedReason: err.message });
  }
});

Real output, for a job that fails all 3 attempts:

[05:10:03.405] 💀 Job 3 exhausted all retries -> moving to DLQ
[05:10:03.407] 🔎 DLQ INSPECTOR received permanently-failed job (original id 3)

Deduplication

BullMQ supports it natively via a deduplication option:

const first = await ordersQueue.add('send-confirmation-email', { orderId: 'order-4' },
  { deduplication: { id: 'order-4-email' } });
const second = await ordersQueue.add('send-confirmation-email', { orderId: 'order-4' },
  { deduplication: { id: 'order-4-email' } });
// first.id === second.id -> the second add was silently merged into the first

Both calls return the same job id `4`: the second add() is recognized as a duplicate and never creates a second job.

2. RabbitMQ (Node.js + amqplib)

Retries with backoff

RabbitMQ has no built-in "attempts" counter or automatic backoff. You build retries yourself using a well-known pattern: a retry queue with a message TTL (time-to-live) that, once expired, dead-letters the message back into the main queue.

await channel.assertQueue('orders-retry', {
  arguments: {
    'x-dead-letter-exchange': 'orders-exchange',
    'x-dead-letter-routing-key': 'orders',
    'x-message-ttl': 2000, // acts as the retry delay
  },
});

The consumer tracks attempts in a custom header and decides where a failed message goes next, either back to the retry queue or to a final DLQ:

const retryCount = msg.properties.headers['x-retry-count'] || 0;
if (retryCount + 1 >= MAX_RETRIES) {
  channel.publish('orders-dlx-exchange', 'orders-dlq', msg.content, { headers: { 'x-retry-count': retryCount + 1 } });
} else {
  channel.publish('orders-retry-exchange', 'orders-retry', msg.content, { headers: { 'x-retry-count': retryCount + 1 } });
}
channel.ack(msg);

The resulting log shows the ~2 second gap between attempts, coming entirely from the retry queue's TTL, and the final message landing in the DLQ inspector on the 3rd failure:

[05:11:21.841] ❌ order order-3 failed attempt 1 - scheduling retry in 2000ms
[05:11:23.848] ❌ order order-3 failed attempt 2 - scheduling retry in 2000ms
[05:11:25.855] 💀 order order-3 exhausted 3 attempts -> publishing to DLQ
[05:11:25.858] 🔎 DLQ INSPECTOR received permanently-failed order

The dead letter queue itself is just orders-dlq, bound to orders-dlx-exchange above: any message the consumer routes there sits untouched until a human looks at it.

Deduplication

RabbitMQ has no native deduplication. You have to track "seen" IDs yourself: in memory for a quick demo, in Redis or a database in production.

if (seenDedupIds.has(dedupId)) {
  channel.ack(msg); // drop the duplicate
  return;
}
seenDedupIds.add(dedupId);

Verified output: 🔁 DEDUP: message with x-dedup-id=order-4-confirmation already processed - dropping duplicate.

3. Amazon SQS (AWS SDK v3)

Retries

SQS retries are driven by the visibility timeout: when a worker receives a message, SQS hides it from other workers for N seconds. If the worker doesn't explicitly delete it in time, it becomes visible again. That's an automatic retry, with no extra code needed:

const { Messages } = await client.send(new ReceiveMessageCommand({
  QueueUrl: mainUrl, AttributeNames: ['ApproximateReceiveCount'],
}));
// if processing fails, simply DON'T call DeleteMessageCommand
// -> the message reappears after the VisibilityTimeout

Dead letter queue

This is fully native and declarative. You attach a redrive policy to the main queue naming the DLQ and how many receives to allow first:

await createQueue('orders', {
  VisibilityTimeout: '2',
  RedrivePolicy: JSON.stringify({ deadLetterTargetArn: dlqArn, maxReceiveCount: 3 }),
});

order-3 is received 3 times, never deleted, and SQS itself (not your code) moves it to the DLQ:

[05:22:49.582] WORKER received order order-3 - ApproximateReceiveCount=1
[05:22:52.412] WORKER received order order-3 - ApproximateReceiveCount=2
[05:22:55.404] WORKER received order order-3 - ApproximateReceiveCount=3
[05:23:00.421] --- FINAL STATE: 1 message(s) sitting in the DLQ ---
[05:23:00.421]   DLQ entry: {"orderId":"order-3","amount":99,"failTimes":99}

Deduplication

Native, but only on FIFO queues (queue names ending in .fifo), using MessageDeduplicationId:

await client.send(new SendMessageCommand({
  QueueUrl: fifoUrl, MessageGroupId: 'orders',
  MessageDeduplicationId: 'order-4-confirmation',
  MessageBody: JSON.stringify({ orderId: 'order-4' }),
}));
// sending the exact same MessageDeduplicationId again within 5 minutes is silently ignored

After sending the same MessageDeduplicationId twice, the queue holds exactly 1 message, not 2.

Common Problems / Errors

1. Jobs seem "stuck" and never retry (BullMQ). Usually caused by not setting attempts at all. The default is 1, so a failed job fails permanently with no retry.
2. Messages processed twice (RabbitMQ/SQS). If a worker crashes after finishing the work but before acking/deleting the message, the broker assumes it failed and redelivers it. The fix is to make job handlers idempotent (safe to run twice), not just to add deduplication.
3. DLQ fills up silently. Without monitoring/alerting on DLQ depth, permanently failed jobs pile up unnoticed for weeks.
4. Visibility timeout too short (SQS). If processing takes longer than the visibility timeout, SQS makes the message visible again while it's still being processed, causing an accidental duplicate run.
5. Deduplication window too short. BullMQ/SQS dedup windows are time-bound (ttl, 5-minute SQS window). A legitimate retry sent by the user just outside that window is treated as a brand-new task, not a duplicate.

Best Practices

• Always set an explicit attempts (or maxReceiveCount). Don't rely on defaults.
• Use exponential backoff with jitter so retries don't all hammer a struggling downstream service at once.
• Make every job handler idempotent. Deduplication reduces duplicates, but it doesn't guarantee zero.
• Alert on DLQ depth, not just on failures. A DLQ nobody looks at is a silent data-loss risk.
• Keep the DLQ's retention period longer than the main queue's, so a slow week doesn't quietly delete evidence.
• Set visibility timeout / job lock duration comfortably longer than your slowest expected processing time.
• Log the job/message ID at every stage. It's the only way to trace one task across producer, retries, and DLQ.

Performance & Security Considerations

Redis and RabbitMQ connections should use TLS and authentication in production. An open Redis or RabbitMQ port with no auth is a common breach vector, and it happens more often than it should. SQS access should be scoped with IAM policies to exactly the queues a service needs, not a blanket sqs:* permission.

Throughput and worker sizing matter too. AWS's published quotas cap FIFO queues at 300 messages/second by default, or 3,000/second with batching; if you don't need strict ordering, a standard SQS queue scales further than that. And worker concurrency should match downstream capacity, not just CPU count: running more workers than your database or payment gateway can handle just moves the bottleneck somewhere else.

Alternatives / Comparison

When Should You Use It?

Choose BullMQ if you're a Node.js shop, already run Redis, and want retries/backoff/dedup with minimal setup.

Choose RabbitMQ if you have multiple services in different languages, need complex routing (fan-out, topic-based routing), and are comfortable operating a broker.

Choose SQS if you're already on AWS and want to avoid running any queue infrastructure yourself. In exchange for zero ops, you accept SQS's own limits: dedup only on FIFO queues, and no built-in backoff.

FAQ

Is BullMQ a message broker like RabbitMQ? No. BullMQ is a library that implements queue behavior on top of Redis; RabbitMQ is a standalone broker server. Both solve the same problem, differently.

Do I need a dead letter queue for every queue? For anything where a failure matters (payments, notifications), yes. For purely best-effort tasks (e.g., analytics pings), it's optional.

Can SQS guarantee message order? Only with a FIFO queue (name ending in .fifo). Standard SQS queues make no ordering guarantee and may occasionally deliver a message more than once.

Is job deduplication the same as idempotency? No. Deduplication stops the same task from being queued/processed twice within a window. Idempotency means your handler produces the same correct result even if it does run twice. You want both.

How many retry attempts should I configure? There's no universal number. 3 to 5 attempts with exponential backoff is a common starting point, tuned based on how quickly the downstream service typically recovers.

Does RabbitMQ support retries at all? There's no single attempts setting like BullMQ has. Instead, the retry-queue-with-TTL pattern shown above is the standard, widely used way developers achieve the same effect.

Trade-offs

A queue does not remove the failure, it just moves where you deal with it. That trade is worth knowing before you commit to one.

You lose the instant confirmation. The customer sees "Order confirmed" before the charge, the email, or the notification has actually happened. If any of those steps fails after that point, you need a separate way to tell the customer, because the request that promised success has already closed.

Ordering is no longer free. BullMQ and RabbitMQ queues are per-queue, best-effort; standard SQS makes no ordering guarantee at all. If job B depends on job A finishing first, you need a FIFO queue or your own sequencing, not just "first in, first out" by default.

Retries make idempotency mandatory, not optional. The moment you add automatic retries or an SQS visibility timeout, the same job can run more than once. A handler that charges a card or sends an email has to tolerate that, or deduplication alone will not save you, as the Common Problems section above shows.

A DLQ is a queue, not a fix. Moving a failed job to a dead letter queue stops it from blocking the main queue, but the job is still failed. Without someone watching DLQ depth, it just becomes a second queue nobody looks at.

It's another moving part to run or pay for. BullMQ needs a Redis instance, RabbitMQ needs a broker you operate, and SQS is one more AWS service with its own IAM policy and its own bill. None of that infrastructure existed before you had a queue.

Debugging a multi-hop job is harder than a synchronous failure. A broken synchronous request fails and tells you why in one stack trace. A broken queued job can fail quietly three hops away from where the user ever sees anything, and the job/message ID is the only thread connecting the two.

None of this is a reason to skip a queue for slow or unreliable work. It is a reason to only reach for one when the work genuinely needs to run outside the request path, and to budget for idempotency and DLQ monitoring from the start rather than after the first incident.

Conclusion

A slow or flaky step shouldn't hold up the user, and a single failure shouldn't be the end of the story. Queue-based architecture fixes both problems by moving the work off to a queue and a worker, backed by retries and a dead letter queue for whatever never recovers. BullMQ, RabbitMQ, and SQS all implement that same core idea, with varying native support for deduplication, but they land in different places on setup effort, control, and hosting. In short: BullMQ suits a Node-and-Redis stack, RabbitMQ suits multi-language systems that need real routing control, and SQS suits teams on AWS who'd rather not run queue infrastructure at all.