# Flaky tests: from 0.5% to 98%

> If each of 800 tests fails for no reason 0.5% of the time, the suite is red on almost every run. Why retry isn't a cure, and what catches flakiness?

- Kind: Infographic
- Published: 2026-10-03
- Based on: https://muhammetsafak.com/engineering/essays/flaky-tests-and-determinism/
- Source: https://muhammetsafak.com/visuals/i/flaky-tests-98-percent-red/
- Language: en-US
- Author: Muhammet Şafak

---
## Panel 1

98%

The bigger the suite, the less reliable

Chance that any test fails on a clean run

A single test that fails for no reason 0.5% of the time is invisible. Put 800 of them together: 1 − 0.995^800 ≈ 98%.

## False red as the test count grows

Probabilities compound

at least one test failing for no reason, %

False red as the test count grows (at least one test failing for no reason, %): 100 tests: 39.4, 200 tests: 63.3, 300 tests: 77.8, 400 tests: 86.5, 500 tests: 91.8, 600 tests: 95.1, 700 tests: 97, 800 tests: 98.2, 900 tests: 98.9, 1000 tests: 99.3.

Computed from the formula: 1 − 0.995^n. Not a measurement; the post only gives ≈ 98% for n = 800.

## Panel 3

Failures aren't independent either

50 ms locally, a timeout under load.

CI runs on shared, throttled hardware; an async test that finishes in 50 ms on your laptop falls into a timeout under load.

## Four patterns that look right

AI didn't invent flakiness, it mass-produced it

-   A fixed sleep To "wait" for async work: time.Sleep(2 \* time.Second), await page.waitForTimeout(2000), cy.wait(2000).
-   Unseeded randomness A different value on every run: Math.random(), rand.Intn(100), a faker with no fixed seed.
-   A brittle selector Nailed to the DOM structure: nth-child, an auto-generated class name. Breaks when the markup shifts a little.
-   A mock for everything The test verifies an interaction, not an outcome.

## All of it is reactive

Retry is a treadmill, not a cure

-   Retry Burns real CI minutes and stretches the feedback loop; you pay twice to learn nothing new.
-   Quarantine A silently growing pile of disabled tests. That pile is debt, and it compounds.
-   Ticket Ages in backlog noise and quietly stops mattering.

## When does it step in?

Catch the anti-pattern, not the failure

|  | Retry, quarantine, ticket | Static scan at the commit boundary |
| --- | --- | --- |
| Timing | After the test has already broken CI | Before the test even runs, in the diff |
| Effect | Hides the symptom, sends you the bill | Flags the known anti-patterns |
| Repeatability | Passes on rerun, then fails again | Same diff, same result; no model, no network |
| Scope | Doesn't make the test deterministic | Stops obvious flakiness before it ever reaches CI |

## Write the test deterministically

Detection is the safety net, design is the real fix

-   Replace fixed waits with condition-based waits. Wait for a fact, not for the clock. (pending)
-   Seed every source of randomness; inject a deterministic clock and ID. (pending)
-   Select by role or test id, not DOM position. (pending)
-   Mock the boundary, not the logic; keep at least one real integration path. (pending)

## Limits of the static gate

When does this pattern stop being the right answer?

-   Won't catch a clever race It catches the obvious anti-patterns, not a race condition buried inside your own code.
-   Keep the rules conservative A noisy gate gets ignored; an ignored gate is worse than none.
-   If the system is flaky A test that is flaky because the system under test isn't deterministic is telling you something about the system.

## Panel 9

A green CI you have to rerun isn't green.

The cheapest place to kill a flaky test is the line it's written on: before it costs you a build.

## On the line it's written

[muhammetsafak.com/engineering/](https://muhammetsafak.com/engineering/)

Source: Engineering, Flaky Tests: Retrying Isn't a Fix
