Skip to content
← All posts
6 min read Dawid Skłodowski

Background jobs with Sidekiq and Redis

Sidekiq processes background jobs with threads and Redis, and it is fast — but the speed comes with rules. A deep-dive on how it works, why your jobs must be small and idempotent, and how to survive retries and concurrency.

Anything slow that a user should not wait for — sending email, resizing an image, calling a third-party API, generating a report — belongs in a background job. In the Ruby world Sidekiq has become the default tool for this, and deservedly: it is fast, efficient, and pleasant to operate. But that efficiency comes from a concurrency model that imposes real rules on how you write jobs, and most Sidekiq pain comes from breaking them without realising. Here is how it works and how to write jobs that behave.

How Sidekiq works: threads and Redis

Two design choices define Sidekiq. First, it uses threads, not processes, for concurrency. Where Resque forks a whole process per job, a single Sidekiq process runs many worker threads — often 10 or 25 — sharing one Ruby VM. That makes it dramatically more memory-efficient: one process with 25 threads uses a fraction of the RAM of 25 forked Resque workers. The catch, which we will come back to, is that threads share memory, so your job code must be thread-safe.

Second, it uses Redis as the job store. When you enqueue a job, Sidekiq serialises it (the worker class name plus its arguments) and pushes it onto a Redis list. Worker threads pop jobs off Redis and run them. Redis is the perfect fit here: it is fast, atomic, and its data structures (lists, sorted sets) map exactly onto queues and scheduled jobs.

class ThumbnailWorker
  include Sidekiq::Worker
  sidekiq_options queue: :images, retry: 5

  def perform(upload_id)
    upload = Upload.find(upload_id)
    upload.generate_thumbnail!
  end
end

ThumbnailWorker.perform_async(upload.id)          # enqueue now
ThumbnailWorker.perform_in(1.hour, upload.id)     # schedule for later

(In a Rails 4.2 app you can also write this as an ActiveJob and point the adapter at Sidekiq — the same engine underneath, with a backend-agnostic API on top.)

Rule 1: pass IDs, not objects

Look closely at that worker — it takes upload_id, not upload. This is the first rule, and it follows directly from Redis being the store. Job arguments are serialised to JSON and sit in Redis until a worker picks them up, possibly seconds or minutes later. Two things break if you pass a whole object:

  • The serialised copy is a stale snapshot taken at enqueue time. By the time the job runs, the record may have changed; you want the worker to load the current state.
  • Complex objects do not round-trip through JSON cleanly anyway.

So pass the id, and let the job find the record when it runs. The worker always operates on fresh data, and the payload sitting in Redis stays tiny.

Rule 2: jobs must be idempotent

This is the rule that, ignored, causes the most damage — and it follows from how Sidekiq guarantees delivery. Sidekiq is at-least-once: it guarantees a job will run, not that it will run exactly once. A worker can die mid-job (deploy, crash, OOM) after doing some work but before reporting success, and Sidekiq will re-run the job. Retries (below) re-run it too. So every job must be safe to run more than once.

# DANGEROUS — runs twice, charges twice
def perform(order_id)
  order = Order.find(order_id)
  PaymentGateway.charge(order.total)
end

# IDEMPOTENT — the second run is a no-op
def perform(order_id)
  order = Order.find(order_id)
  return if order.charged?
  order.with_lock do
    return if order.charged?
    PaymentGateway.charge(order.total, idempotency_key: "order-#{order.id}")
    order.update!(charged: true)
  end
end

The techniques: check whether the work is already done before doing it, use a database lock to close the race between concurrent runs, and lean on the external service’s own idempotency key when it offers one. For a thumbnail this barely matters; for anything that moves money, sends a message, or has side effects in the outside world, idempotency is not optional — it is the difference between a robust system and one that double-charges customers on your next bad deploy.

Rule 3: respect retries

By default Sidekiq retries a failed job, with exponential backoff, up to ~25 times over about three weeks before giving up to the dead set. This is a feature — a transient failure (a third-party API blip, a deadlock) fixes itself on retry — but it interacts sharply with idempotency. A job that is almost idempotent but charges on the happy path will charge again on every retry. Tune the policy per worker:

sidekiq_options retry: 5            # cap retries for this worker
sidekiq_options retry: false        # never retry (only if truly safe to drop)

Reserve retry: false for jobs where a missed run is harmless or re-triggered elsewhere. For everything else, set a sensible cap and make sure the job is genuinely idempotent so retries are safe rather than dangerous. And watch the dead set — jobs that exhausted their retries are sitting there telling you something is broken.

Rule 4: thread safety is on you

Because workers are threads in one process, anything they share must be thread-safe. In practice this rarely bites modern Rails code — ActiveRecord’s connection pool is thread-safe, and most gems are — but it bites hard when it does. The usual culprits:

  • Mutable class-level state. A @@counter or a memoized class variable that jobs write to is a race condition waiting to happen.
  • Non-thread-safe libraries. Some older gems keep global mutable state; check before using one inside a worker.
  • Connection pool sizing. Your database pool must be at least as large as your Sidekiq concurrency, or threads will queue waiting for a connection. Set the pool to match (or exceed) the worker count.

Operating it well

A few practices that keep Sidekiq healthy in production:

  • Separate queues by purpose and priority, and weight them so a flood of low-priority jobs (say, analytics) cannot starve urgent ones (password-reset emails). sidekiq -q critical,2 -q default processes critical twice as often.
  • Keep jobs small and single-purpose. A job that does five things is five things that must all be idempotent and all retry together. Smaller jobs retry cleanly and parallelise better.
  • Use the Web UI. Sidekiq’s dashboard shows queue depths, retries, and the dead set — mount it (behind auth) and actually look at it; a growing queue or a filling dead set is an early warning.
  • Make enqueuing cheap and the job do the work. Controllers should enqueue and return; all the slow work happens in the worker.

Verdict

Sidekiq is the right default for background jobs in Ruby: the threaded model makes it fast and memory-efficient, and Redis gives it a solid, observable backing store. But its speed is inseparable from its contract — at-least-once delivery, threaded concurrency, automatic retries — and that contract is really one demand on you: write small, idempotent, thread-safe jobs that take IDs and reload their data. Internalise that and Sidekiq is a workhorse you can trust with the most important side effects in your system. Ignore it and it will, eventually, run something twice at the worst possible moment.

#ruby #rails #redis #sidekiq