Flaky Tests: Fix, Quarantine, or Delete

Every team arrives at the same place eventually. The suite fails, somebody re-runs it, it passes, the pull request merges. Nobody investigates, because investigating costs an afternoon and re-running costs thirty seconds.

Do that often enough and the suite stops meaning anything. Not gradually — there is a threshold, and past it people stop reading failures at all. A red build becomes a thing you re-run rather than a thing you read, and at that point a genuine regression ships with a red build nobody looked at.

The question is not how to eliminate flakiness. It is what to do with each flaky test, and there are exactly three answers.

First: is it actually flaky?

Worth ten minutes, because two things get called flaky and only one of them is.

A flaky test passes and fails on identical code and identical inputs. The non-determinism is in the test or its environment.

A test that fails only in CI is not flaky — it is deterministic, and you are running it in two different environments. Same code, different machine, consistent result on each. That has a different cause set entirely and its own page: passes locally, fails in CI.

The distinguishing question: does it fail intermittently in the same environment? Run it fifty times in CI and count.

# Crude and effective. Run the one test repeatedly and count failures.
for i in $(seq 1 50); do
  <your runner> path/to/the.test 2>/dev/null || echo "FAIL on run $i"
done

Fifty green runs means it is probably environmental rather than flaky. A scattering of failures means it is genuinely non-deterministic and the rest of this applies.

The three answers

Every flaky test gets exactly one of these, and the decision should take minutes rather than being deferred indefinitely.

Fix it

When: the test covers something that matters, and the flakiness is in the test rather than the system.

Most flakiness comes from a short list:

• Time. A test asserting on now() that fails at midnight, month end, or in a different timezone. Freeze the clock.
• Order. A test that passes alone and fails in the suite is reading state another test left behind. Run with a randomised seed to find the pair.
• Concurrency. A shared fixture, a database row, a port, a temp file two workers both want. If it is a port, the failure often surfaces as port is already allocated rather than as an assertion.
• Waiting on the wrong thing. A fixed sleep that is long enough on your laptop and not on a loaded runner. Wait for the condition, not for a duration.
• Ordering assumptions. Asserting on a list the database returned without an ORDER BY.

The last one is worth dwelling on, because it is the most instructive. A test that asserts on unordered results is not really flaky — it is correct about a non-deterministic system and asserting as though it were deterministic. Sometimes the fix is in the test; sometimes the test has found something worth knowing about the code.

Quarantine it

When: it matters, you cannot fix it today, and it is blocking people right now.

Quarantine means the test still runs and no longer blocks the merge. It is a legitimate move and it has exactly one requirement: it must expire.

# The quarantine entry that works
test: checkout_flow_completes
quarantined: 2026-09-18
owner: payments-team
issue: ENG-4471
expires: 2026-10-16          # four weeks

A quarantine with an owner and a date is a decision. A quarantine without them is a deletion that still consumes CI minutes and produces output nobody reads — which is worse than deleting it, because the suite looks more thorough than it is.

Review the list on a schedule. If it only grows, quarantine has become your default answer and the suite is quietly hollowing out.

Delete it

When: it does not cover anything the rest of the suite misses, or it has been quarantined twice and nobody has fixed it.

This is the underused option and people resist it, because deleting a test feels like reducing coverage. Consider what is actually being lost. A test that is skipped, or that everyone re-runs past, provides no signal — and it does cost: CI minutes, maintenance on every refactor, and the credibility of every other failure in the suite.

A deleted test is honest about covering nothing. A permanently quarantined test is dishonest about the same thing.

Delete it when: the behaviour is covered elsewhere, or it tests an implementation detail that changes often, or nobody can say what regression it would catch.

Making the decision

The decision tree is short.

Does it guard something that would be expensive to break? No → delete it. Yes → continue.

Can you fix it this week? Yes → fix it. No → quarantine with an owner and a four-week expiry.

Has it already been quarantined once and come back? Then it is not getting fixed. Either somebody owns it properly now, or delete it and note what is no longer covered.

That third question is the one that keeps the system honest. Without it, quarantine becomes a way of never deciding.

Measure it, or it will not improve

You cannot manage this by impression. Two numbers are enough:

Flake rate — the proportion of pipeline runs that fail and then pass on re-run with no code change. Trending this is more useful than any absolute value.

Top offenders — which tests account for the failures. This is almost always concentrated: a handful of tests generate most of the re-runs, and fixing those few changes the experience disproportionately.

Most CI systems can report both from their own run history. If yours cannot, a few lines parsing the job log will do, and knowing which five tests cost you the most is worth considerably more than the effort.

What not to do

Automatic retries on everything. Tempting, and it converts a visible problem into an invisible one — the suite now hides genuine intermittent bugs in your product, which are exactly the bugs worth finding. Narrow, logged retries at a known-unstable boundary can be defensible. A blanket retry policy is a way of not knowing.

Adding sleeps. A sleep turns a deterministic failure into a probabilistic one and makes the suite slower forever. Wait for the condition.

Blaming CI. The runner is slower and more constrained than your laptop, which is why the race shows up there. The race was always in the test. Worth ruling out the genuinely environmental causes first though — a runner killed for memory shows up as exit code 137, not as flakiness.

Deferring the decision. The cost of a flaky test is not the failed run — it is the erosion of everyone's willingness to read a failure. That cost accrues from the day it first flakes.

The short version

Confirm it is flaky rather than environmental — same environment, intermittent result. Then pick one of three, today: fix it, quarantine it with an owner and an expiry, or delete it. Track the flake rate and the top offenders, because a handful of tests cause most of it.

The thing being protected is not coverage. It is whether anyone still believes a red build.