Image pipelines for big product catalogs

A product catalog is, visually, mostly images — and images are where a naive e-commerce app quietly falls over. Each product wants a thumbnail in the grid, a medium image on the listing, a large one on the detail page, a zoom version, and a square crop for the cart. Multiply that by tens of thousands of products and a few uploads a day, and “just resize the image” becomes an infrastructure problem that can melt your web servers and slow every page. Here is how to build an image pipeline that scales, drawn from running catalogs with serious image volume.

The cardinal rule: keep the original, derive the rest

The first principle is to treat the uploaded file as a master you never modify, and generate every displayed size as a derivative from it. Store the original at full quality, untouched, and produce thumbnails, medium, large, and crops from that source.

Why this matters so much: requirements change. Today you show a 300px thumbnail; next quarter the designer wants 360px and a new square crop for a redesigned cart. If you discarded the original and only kept the sizes you needed at upload time, you are stuck — you cannot up-res a 300px image to 360px without it looking terrible. Keep the master and regenerating the entire catalog at a new size is a background job you can run anytime. Discard it and a redesign means re-uploading tens of thousands of product photos.

# CarrierWave: the original is kept; versions are derived from it
class ProductImageUploader < CarrierWave::Uploader::Base
  storage :fog               # store on S3, not the app server

  version :thumb  { process resize_to_fill: [300, 300] }
  version :medium { process resize_to_fit:  [800, 800] }
  version :large  { process resize_to_fit:  [1600, 1600] }
end

(In 2016 the common tools are CarrierWave or Paperclip over ImageMagick; the principles outlive whichever you pick.)

Generate variants in the background, never in the request

The second principle, and the one that saves your servers: image processing does not belong in a web request. Resizing a large photo into five variants with ImageMagick is CPU- and memory-heavy and takes seconds. Do that inline when a product is saved and you tie up a web worker for seconds per upload, and a bulk import of a thousand products will take your whole site down as workers starve.

Push it to a background job. The upload stores the original immediately and enqueues the variant generation; a worker does the heavy lifting off the request path:

class Product < ApplicationRecord
  after_commit :enqueue_image_processing, on: [:create, :update], if: :image_changed?

  def enqueue_image_processing
    GenerateImageVariantsJob.perform_later(id)
  end
end

This also lets you scale image processing independently — run more workers (or beefier ones) for the CPU-bound resizing without touching your web tier. The page that uploads a product returns instantly; the variants appear moments later. A bulk import becomes a flood of background jobs that drain at the workers’ pace instead of a denial-of-service against your own site.

Serve from object storage and a CDN, not your app

The third principle: your application servers should never serve image bytes. Two moves:

• Store the files in object storage (S3), not on the app server’s disk. App-server disk does not scale, does not survive a redeploy on ephemeral infrastructure, and is not shared across multiple servers. Object storage is durable, effectively infinite, and accessible from every worker and web node — the same “container is disposable, data is durable” lesson, applied to uploads.
• Put a CDN in front. Product images are static, identical for every visitor, and requested constantly — the textbook case for a CDN. Serving them from edge caches close to the user makes pages dramatically faster and removes that entire load from your origin. Your app generates and stores the variant once; the CDN serves it a million times.

Cache-bust by including a hash or version in the filename so a changed image gets a new URL and the CDN does not serve a stale one.

Eager vs lazy variant generation

A real design decision: do you generate every variant up front (eager), or on first request (lazy)? Both have a place:

• Eager (generate all sizes when the image is uploaded, via the background job) gives predictable, instant serving — every variant exists before anyone asks for it. The cost is storage for sizes that may rarely be viewed, and a big regeneration job when you add a size.
• Lazy / on-the-fly (generate a variant the first time that size is requested, then cache it) saves storage and adapts instantly to new sizes, but the first request for each size pays the processing cost, and you must guard against abuse (someone requesting thousands of arbitrary dimensions to hammer your processor). On-demand image services (or a imgproxy-style resizer behind the CDN) implement this pattern.

For a catalog with a known, fixed set of sizes, eager generation in the background is usually simplest and most predictable. For systems with many or unpredictable sizes, lazy-with-caching wins. We have used both; the deciding factor is whether your set of sizes is small and stable.

Optimise the bytes, not just the dimensions

Resizing is only half the job; the file also needs to be light. A correctly-sized image that is 800KB still makes the page slow. Strip metadata, choose sensible compression quality (JPEG quality around 80 is usually indistinguishable from 100 at a fraction of the size), and serve appropriately for the context. Progressive JPEGs render perceptually faster. These optimisations, applied in the same background job that resizes, are often a bigger real-world speed win than the resize itself.

Verdict

Image handling is the part of an e-commerce build that looks trivial and quietly becomes an infrastructure problem at catalog scale. The pipeline that holds up rests on four principles: keep the untouched original and derive every size from it, so a redesign is a background job not a re-upload; generate variants in background workers, never in the web request, so a bulk import doesn’t take the site down; store in object storage and serve through a CDN, so your app never ships image bytes; and optimise the file size, not just the dimensions. Build those in from the start and a catalog of a hundred thousand products serves images fast and cheaply. Bolt image handling on naively and it becomes the thing that wakes you up when the next big import lands.