CI/CD pipelines that don't flake

A CI/CD pipeline has one job: tell you, reliably, whether your code is safe to ship. The keyword is reliably. A pipeline that is slow, or — far worse — flaky, fails at that job in a particularly corrosive way: it trains your team to distrust it. Once “the build is red again, just re-run it” becomes a reflex, a red build means nothing, and a real failure sails through because nobody believes the signal. A flaky pipeline is genuinely worse than no pipeline, because it manufactures false confidence. Here is how to build one people actually trust.

The enemy: flaky tests

The single biggest threat to a pipeline’s credibility is the flaky test — one that passes and fails on the same code, nondeterministically. Each flaky test is a small dose of poison: it teaches everyone that red does not necessarily mean broken, and that lesson, once learned, applies to the real failures too. Treat flaky tests as incidents, not annoyances. The common causes and their fixes:

• Order dependence. A test that passes only because an earlier one left state behind fails when the order changes. Run your suite in random order (--order random) to surface these, and make each test set up and tear down its own world. (This is the same discipline parallel testing forces.)
• Time and dates. Tests that depend on the current time, or assume “now” does not cross a day boundary mid-run, fail intermittently. Freeze time in tests (timecop/travel_to) rather than using the real clock.
• Async and race conditions. Feature tests that assert before an async action completes flake constantly. Wait for the condition (Capybara’s matchers wait), never sleep an arbitrary amount and hope.
• Shared external state. Two tests, or two parallel workers, touching the same file, port, Redis db, or VCR cassette. Namespace and isolate them.

The rule we hold to: when a test flakes, you fix it or delete it — you do not re-run and move on. A quarantined or fixed flaky test keeps the suite honest; a tolerated one rots it.

Make builds deterministic

Flakiness is just nondeterminism leaking into your build, so the broader principle is to make the whole pipeline deterministic — the same inputs always produce the same result:

• Pin everything. Lock gem versions (Gemfile.lock), Node versions, the base Docker image, system package versions. A build that pulls “latest” anything is a build that can break without a code change — the most maddening kind of red.
• Build in a clean, reproducible environment. This is where the Docker work pays off: running CI in the same container you develop in eliminates “works on my machine, fails in CI” caused by environment drift. The build environment is itself pinned and reproducible.
• No dependence on external services in unit tests. A test that hits a real third-party API flakes whenever that API is slow or down. Stub external calls; test the integration separately and deliberately.

A deterministic build means a red build is always a real signal — which is the entire point.

Make it fast, because slow erodes trust too

A correct pipeline that takes 40 minutes is also a failure, in a subtler way: people stop waiting for it, merge before it finishes, and context-switch away, so the feedback loop that CI exists to provide is broken. Speed is a feature:

• Parallelise the test suite across cores and CI nodes, balanced by runtime (covered in the parallel-testing post) — this is usually the biggest single win.
• Cache aggressively. Cache installed gems, node_modules, and compiled assets between runs so every build is not paying full setup cost. Cache Docker layers.
• Fail fast. Run the quick, cheap checks first (linting, fast unit tests) so an obvious failure reports in seconds, not after the full slow suite. Run the expensive browser tests last.
• Only build what changed where you can, in a larger codebase.

The target is a pipeline fast enough that people wait for it — that is when CI actually shapes behaviour.

From CI to CD: deploying safely

Continuous integration tells you the code is good; continuous delivery/deployment gets it to production safely and automatically. The pipeline extends past the tests into deploy, and the disciplines that keep deploys safe build on earlier posts:

• Every commit to main is potentially shippable — which is what trunk-based development and feature flags (covered earlier) make possible: incomplete work hides behind flags, so main stays releasable.
• Automate the deploy so it is repeatable and boring — the same steps every time, no manual ritual to get wrong, with the atomic-release and easy-rollback properties the Capistrano post described.
• Roll out gradually and watch. Deploy to staging, then production behind a flag or to a fraction of traffic (canary), and watch the metrics that matter (the monitoring post’s golden signals). If error rate or latency spikes, roll back — fast and automatically where you can.
• Make rollback trivial. The most important safety property of a deploy pipeline is that you can undo a bad release in seconds. If rollback is hard, every deploy is scary; if it is one command (or automatic on a failed health check), deploys become routine.

Verdict

A CI/CD pipeline is worth exactly as much as the team’s trust in it, and trust is destroyed by flakiness and slowness. So the priorities, in order: hunt flaky tests as incidents (fix or delete, never re-run-and-shrug), make the whole build deterministic (pin everything, build in a clean container, stub external services) so red always means broken, and make it fast (parallelise, cache, fail fast) so people actually wait for it. Then extend that trustworthy foundation into deploy — automated, gradual, watched, and instantly reversible. Get this right and the pipeline becomes what it should be: a fast, honest signal that lets you ship small changes confidently many times a day. Get it wrong and you have built an expensive random number generator that everyone has learned to ignore.