Bull - Queue system for handling distributed jobs and messages in Node

26 Jul, 2019

Bull is the fastest, most reliable, Redis-based queue for Node. I have been using it for years to handle async jobs and schedule messages

There are many queueing systems out there. Each one of them is different and was created for solving certain problems: ActiveMQ, Amazon MQ, Amazon Simple Queue Service (SQS), Apache Kafka, Kue, Message Bus, RabbitMQ, Sidekiq, Bull, etc.

What Message Queue is best fit for a Node.js application? You asked! I came across implementations like Kue and Bull. They are redis backed and seem like good solutions. Have you guys tried any of these? I've been using Bull for years to handle async jobs and schedule messages and quite satisfied with it!

Bull

Bull is a Node library that implements a fast and robust queue system based on redis.

logo

Although it is possible to implement queues directly using Redis commands, this library provides an API that takes care of all the low-level details and enriches Redis basic functionality so that more complex use-cases can be handled easily.

If you are new to queues you may wonder why they are needed after all. Queues can solve many different problems in an elegant way, from smoothing out processing peaks to creating robust communication channels between microservices or offloading heavy work from one server to many smaller workers, etc.

Features

  • Minimal CPU usage due to a polling-free design.
  • Robust design based on Redis.
  • Delayed jobs.
  • Schedule and repeat jobs according to a cron specification.
  • Rate limiter for jobs.
  • Retries.
  • Priority.
  • Concurrency.
  • Pause/resume—globally or locally.
  • Multiple job types per queue.
  • Threaded (sandboxed) processing functions.
  • Automatic recovery from process crashes.

Comparison

Since there are a few job queue solutions, here is a table comparing them:

  • Kue: A priority job queue backed by redis, built for node.js
  • Bee Queue: A simple, fast, robust job/task queue for Node.js, backed by Redis.
  • Agenda: Lightweight job scheduling for Node.js
Feature (Jul 2019) Bull Kue Bee Agenda
Backend redis redis redis mongo
Priorities
Concurrency
Delayed jobs
Global events
Rate Limiter
Pause/Resume
Sandboxed worker
Repeatable jobs
Atomic ops
Persistence
UI
Optimized for Jobs / Messages Jobs Messages Jobs

A Job's Lifecycle

In order to use the full potential of Bull queues, it is important to understand the lifecycle of a job. From the moment a producer calls the add method on a queue instance, a job enters a lifecycle where it will be in different states, until its completion or failure (although technically a failed job could be retried and get a new lifecycle).

Diagram showing job statuses

When a job is added to a queue it can be in one of two states, it can either be in the "wait" status, which is, in fact, a waiting list, where all jobs must enter before they can be processed, or it can be in a "delayed" status: a delayed status implies that the job is waiting for some timeout or to be promoted for being processed, however, a delayed job will not be processed directly, instead it will be placed at the beginning of the waiting list and processed as soon as a worker is idle.

The next state for a job I the "active" state. The active state is represented by a set, and are jobs that are currently being processed, i.e. they are running in the process function explained in the previous chapter. A job can be in the active state for an unlimited amount of time until the process is completed or an exception is thrown so that the job will end in either the "completed" or the "failed" status.

Bull Arena

An intuitive Web GUI for Bee Queue and Bull. Built on Express so you can run Arena standalone, or mounted in another app as middleware. Having following features:

  • Check the health of a queue and its jobs at a glance
  • Paginate and filter jobs by their state
  • View details and stacktraces of jobs with permalinks
  • Restart and retry jobs with one click

Same series:

Tech Explained