Background Jobs and Message Queues: BullMQ, Celery, and Laravel Queues in Practice

Background Jobs and Message Queues: BullMQ, Celery, and Laravel Queues in Practice

Web requests are a poor place for slow work: parsing large files, sending emails, generating thumbnails, and calling third-party APIs all make responses slow or time out. A task queue moves this work to the background, processed by dedicated worker processes, so web requests return instantly and user experience improves. Based on the official BullMQ, Celery, and Laravel Queues documentation, this guide walks through the usage and trade-offs of three mainstream approaches.

Core Concepts: Queues, Jobs, and Workers

A task queue is fundamentally a producer-consumer pattern: producers (usually web requests) push "work units" onto a queue, and worker processes listen and execute them.

• Job/Task: a unit of work to execute, containing data and logic.
• Queue: where pending jobs are stored, supporting FIFO, priorities, and delayed execution.
• Worker: a long-running process that pulls jobs from the queue and runs them, horizontally scalable.

The key benefits: smoothing peaks (handling traffic spikes), asynchrony (instant web responses), and reliability (jobs can retry and failures can be traced).

BullMQ: Redis-Based Queues for Node.js

BullMQ is a Redis-based Node.js queue library that the docs describe as "fast and robust." Core features:

• Distributed job execution on Redis; horizontal scaling is simple — add workers to process in parallel.
• FIFO and LIFO, priorities, delayed jobs, and scheduled/repeatable jobs via cron.
• Automatic retries on failure, per-worker concurrency settings, and automatic recovery from process crashes.
• "At least once" delivery semantics: duplicates are possible in rare cases, so business logic must be idempotent.

import { Queue, Worker } from 'bullmq';

const queue = new Queue('emails');
await queue.add('send', { to: '[email protected]' });

const worker = new Worker('emails', async job => {
  await sendEmail(job.data);
});

BullMQ is a common partner for Node.js Express or NestJS backends, and pairs naturally with Redis Caching and Backend Performance Guide.

Celery: Task Queues in the Python Ecosystem

Celery is the most popular distributed task queue in Python; the docs define a task queue as "a mechanism to distribute work across threads or machines." It mediates between clients and workers via a broker (message middleware). RabbitMQ and Redis are feature-complete brokers, with support for SQS, SQLite (local development), and more.

from celery import Celery

app = Celery('tasks', broker='amqp://guest@localhost//')

@app.task
def send_email(to):
    # send email
    pass

Celery features include result stores (Redis, Memcached, databases, etc.), workflow orchestration (group/chain/chord), time and rate limits, and monitoring event streams. The docs note a single process can process millions of tasks a minute with sub-millisecond round-trip latency under optimized RabbitMQ settings. Python backends often integrate it with FastAPI, Django, or Flask.

Laravel Queues: A Unified Queue API for PHP

Laravel's built-in queue system provides a unified API across backends: database, Redis, Amazon SQS, Beanstalkd, and even synchronous execution (for development/testing). The core concept is "connections vs. queues": one connection can host multiple queues used for tiering or prioritization.

ProcessPodcast::dispatch($podcast);                    // default queue
ProcessPodcast::dispatch($podcast)->onQueue('emails'); // specific queue

Start a worker with php artisan queue:work --queue=high,default --tries=3; in production use Supervisor to keep worker processes alive and auto-restart them. Laravel also provides job chaining, batching, unique jobs (ShouldBeUnique), a failed-jobs table, and the Horizon dashboard. See Laravel Backend Development Guide for details.

RabbitMQ and a Selection Comparison

RabbitMQ is itself a general-purpose message broker requiring its own deployment; it suits cross-language, complex-routing scenarios. If you already run Redis, BullMQ or Laravel Queues (Redis driver) is simpler.

Engineering Practices

• Make jobs idempotent: queues are mostly "at least once" delivery, so re-execution must not cause side effects.
• Configure retries with backoff: transient failures (third-party API hiccups) auto-retry with exponential backoff; permanent errors should fail fast into a dead-letter path.
• Monitor and alert: watch queue depth, failure rates, and worker liveness; Laravel uses Horizon/Telescope, BullMQ exposes metrics.
• Deploy gracefully: during releases, let workers finish current jobs before restarting so no work is lost.

A Real-World Case: Thumbnail Generation

Let's tie the concepts together with a real flow. A user uploads a 5MB image; the web request does only two things — stores the file in object storage and enqueues a job on the thumbnails queue — then returns "processing" immediately. A worker pulls the job, crops 128/512/1024 sizes, writes them back, and finally notifies the frontend through a callback.

// BullMQ: configure retries and backoff
const worker = new Worker('thumbnails', async job => {
  await generateThumbnails(job.data.imageId);
}, {
  concurrency: 8,                                  // 8 jobs in flight per worker
  attempts: 5,                                     // auto-retry up to 5 times
  backoff: { type: 'exponential', delay: 2000 },   // exponential backoff
});

Handle failures in layers: transient errors (like object-storage hiccups) go to retries; after repeated failures cross a threshold, send the job to a dead-letter queue for manual review instead of retrying forever and burning CPU. Celery does this with retry_backoff=True and max_retries; Laravel uses the tries and backoff methods — same idea.

Sizing Your Workers

More workers is not always better. A useful rule of thumb: concurrency ≈ target throughput × per-job duration. If thumbnails average 1.2 seconds and you want 120 jobs per minute, you need roughly 120 × 1.2 / 60 ≈ 2.4 units of concurrency, so 3-4 workers are enough — then scale with queue-depth monitoring. Watch whether "queue backlog is growing" instead of blindly adding machines.

Reference: BullMQ docs https://docs.bullmq.io/, Celery user guide https://docs.celeryq.dev/en/stable/userguide/tasks.html, Laravel queues docs https://laravel.com/docs/queues

16IDC perspective

Background job queues are the dividing line between "instant web responses" and "reliable processing." For indie developers, start small: move obviously slow work like email and thumbnails into a queue, and you will often see immediate response-time improvements. More complete backend engineering practices live in the Backend Integration category.

Source: https://docs.bullmq.io/