Script Valley
System Design: APIs, Caching & Scalability
Message Queues and Async ProcessingLesson 5.3

How BullMQ and Redis-backed job queues work

BullMQ architecture, job states, worker concurrency, job priority, scheduled jobs, job events, Redis Streams, failed job handling

How BullMQ and Redis-backed job queues work

BullMQ job lifecycle

BullMQ Basics

BullMQ is a Node.js job queue built on Redis Streams. Jobs move through states: waiting, active, completed, or failed. Workers claim jobs atomically — no two workers process the same job simultaneously.

import { Queue, Worker } from 'bullmq';

// Producer
const emailQueue = new Queue('emails', { connection: { host: 'redis' } });
await emailQueue.add('send-welcome', { userId: 42 }, {
  attempts: 3,
  backoff: { type: 'exponential', delay: 2000 }
});

// Consumer
const worker = new Worker('emails', async (job) => {
  await sendEmail(job.data.userId);
}, { connection: { host: 'redis' }, concurrency: 5 });

Concurrency and Priority

Set concurrency per worker to process multiple jobs in parallel. Add priority (lower number = higher priority) so critical jobs jump the queue. Set attempts and backoff for automatic retry with exponential delay on failure.

Scheduled and Delayed Jobs

// Run after 10 minutes
await queue.add('cleanup', data, { delay: 600000 });

// Repeating job via cron
await queue.add('daily-report', data, {
  repeat: { cron: '0 8 * * *' }
});

BullMQ replaces many cron setups with a more reliable, observable system — job history, failure tracking, and retry logic are built in.

Up next

Event-driven architecture: pub/sub pattern explained

Sign in to track progress