Flaky tests | ComplexQA Wiki

A flaky test is an automated test that both passes and fails on the same code: the product, the test and the configuration do not change, but the outcome does. A failure of such a test does not show whether the product has a defect, and a pass does not show that it has none. Flaky tests are also called non-deterministic or intermittent tests.

The term comes from engineering practice. In 2011 Martin Fowler described the problem in the article "Eradicating Non-Determinism in Tests". In 2016 the Google Testing Blog published "Flaky Tests at Google and How We Mitigate Them" by John Micco, which defines a flaky result as a test showing both a passing and a failing result with the same code. The first large empirical study of the causes, "An Empirical Analysis of Flaky Tests" by Luo, Hariri, Eloussi and Marinov, was presented at the FSE conference in 2014. The ISTQB Advanced Level Test Automation Engineering syllabus (version 2.0, 2024) does not use the word "flaky": it speaks of test cases that do not give a reliable result, for example because of race conditions, and of intermittent failures. Test runners use "flaky" as a status: Playwright and Maven Surefire mark a test that failed and then passed on a retry as flaky.

The problem flaky tests cause

An automated test is useful when its failure means a problem in the product. A flaky test breaks this link. After several false failures the team starts to rerun the pipeline instead of reading the failure. Fowler notes the consequence: once developers learn to ignore the failures of one test, they also ignore the real failures of the others.

The scale of the problem in a large code base was published by Google in 2016:

about 1.5 percent of all test runs reported a flaky result;
almost 16 percent of tests had some level of flakiness;
about 84 percent of the observed transitions from pass to fail in post-submit testing involved a flaky test.

For a project with 1000 tests and a flake rate of 1.5 percent, about 15 tests would fail in a run for reasons unrelated to the change. Every such failure blocks a merge, costs time for analysis and lowers trust in the results of the whole suite.

Causes of flakiness

A test is flaky when its outcome depends on something that the test does not control. The usual sources of such dependencies are described below.

Asynchronous waits and timing

The test performs an action and checks the result before the asynchronous operation has finished: a request to the server, a page render, a message in a queue. A fixed pause in the test hides the problem as long as the operation is faster than the pause. On a loaded CI agent the operation takes longer, and the test fails. The ISTQB CTAL-TAE syllabus names hard-coded waits as a root cause of many test failures.

Test order dependency and shared state

One test leaves state that another test relies on: a record in the database, a file, a value in a cache, a static variable. The second test passes when it runs after the first one and fails when the order changes, for example when tests are distributed between parallel workers. Fowler recommends keeping tests isolated, so that the execution of one test does not affect any other.

Test data

Tests that share accounts or records interfere with each other. Two tests that rename the same customer in parallel see each other's changes. A test that expects an empty list fails when another test has added an element. Data that expires, such as a token, a trial period or a reserved slot, makes a test depend on the date of the run.

Environment and external services

The test depends on a network, a third-party API, a mail server, a shared test environment. A timeout or a temporary error of such a service fails the test without any change in the product. Fowler suggests replacing unstable remote services with test doubles, stand-ins that behave like the real service. Differences between agents also play a role: the number of cores, the available memory, the screen resolution, installed fonts.

Concurrency

The product or the test itself has several threads or processes, and the result depends on the order in which they run. A race condition shows up only with some interleavings of operations. Such defects are often real defects of the product, not only of the test.

Randomness, time and time zones

Random input data, random identifiers and iteration over unordered collections give a different execution on every run. Tests that read the system clock fail at the end of the month, at midnight, on a change of daylight saving time or on an agent in another time zone. Fowler recommends always wrapping the system clock, so that a test can substitute it.

Resource leaks

A test does not release a connection, a file handle, a port, a browser process or memory. Subsequent tests fail when the resource runs out, and the failure appears in a test that has nothing to do with the leak. Fowler describes a resource pool of size one as a way to make the first test after a leak fail and point to the cause.

Empirical data

Luo, Hariri, Eloussi and Marinov studied commits in the Apache Software Foundation repositories whose messages mention intermittent or flaky tests. They found 486 commits that likely fix distinct flaky tests in at least 51 projects and analyzed 201 of them in detail. For 161 commits the root cause could be classified. The distribution of root causes:

Root cause	Commits	Share of 161
Asynchronous wait	74	45%
Concurrency	32	20%
Test order dependency	19	12%
Resource leak	11	7%
Network	10	6%
Time	5	3%
Input and output	4	2%
Randomness	4	2%
Floating point operations	3	2%
Unordered collections	1	1%

The study also reports how the tests were fixed. 42 of the 74 asynchronous wait tests were fixed with an explicit wait for a condition; 20 were "fixed" by adding or extending a sleep, which only reduced the chance of failure. All fixes of order-dependent tests removed the flakiness, most often by setting up or cleaning shared state. In 38 of 161 cases (24 percent) the fix also changed the code under test, and in most of these cases it fixed a real defect. 126 of the 161 tests were flaky from the moment they were written.

Detecting flaky tests

Reruns on the same commit

The direct way to show flakiness is to run a test several times on the same commit without any changes. If the outcomes differ, the test is flaky by definition. Playwright runs every test several times with the option --repeat-each, and a failed job can be repeated with --last-failed, which runs only the tests that failed. A rerun confirms flakiness but does not find the cause.

History of runs

Every run of an automated suite produces a result for each test. When results are kept as a history, flakiness becomes visible as flips between passed and failed without a relevant change:

Test "checkout > pay with saved card", branch main

commit   run    result
a41f2c   #812   passed
a41f2c   #813   failed    rerun of the same commit
b7d903   #814   passed
c1e5aa   #815   passed
c1e5aa   #816   failed    rerun of the same commit
d90b4e   #817   passed

Two failures on commits that also passed are a sign of a flaky test, not of a regression. A test that fails on every run starting from one commit points to a change. Results of runs and their history in a test management system are described in the article Test run. Metrics derived from such history include the share of runs with a flip and the number of flips per test over a period.

Dedicated jobs

Some causes appear only under particular conditions. Teams run separate jobs to expose them:

a nightly job that repeats every test many times;
a job that runs tests in a different or random order, to find order dependencies;
a job that runs each test in isolation, to separate the test from its neighbors;
a job on an agent with limited resources, where timing problems appear more often.

The CTAL-TAE syllabus recommends executing a few isolated tests when analyzing an unexpected result and notes that monitoring system resources can give clues to the root cause of intermittent failures.

Quarantine

Quarantine is the practice of moving a flaky test out of the suites whose results gate a merge or a release, while the test continues to run separately. Fowler describes it as keeping non-deterministic tests in a different suite from the healthy tests. The Google post describes a tool that quarantines tests with too high flakiness and files a bug for each of them; the same post warns that this can hide real defects.

A quarantine policy usually defines:

entry criterion: for example, two failures and passes on the same commit within a week;
owner: the team responsible for the test, and a defect or task for the fix;
limit: Fowler suggests limiting the number of tests in quarantine (for example, eight) or the time a test may stay there (for example, a week);
exit criterion: the fix is merged and the test passes a series of repeated runs;
visibility: quarantined tests still run and their results are reported, so that a real regression in them is noticed.

A test that stays in quarantine without an owner no longer checks anything. Removing it is then an explicit decision rather than a side effect.

Retries and their risks

A retry reruns a failed test automatically in the same job. Test runners support it directly. In Playwright the option retries sets the maximum number of retries, and a test that failed first and passed on a retry gets the status "flaky":

import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  use: {
    trace: 'on-first-retry',
  },
});

The trace is recorded on the first retry, so a flaky failure leaves data for analysis. Since version 1.52 the option failOnFlakyTests (on the command line --fail-on-flaky-tests) makes the run fail if any test is flaky. In pytest the plugin pytest-rerunfailures adds the options --reruns, --reruns-delay and --only-rerun, the last of which retries only failures that match a pattern:

pytest --reruns 2 --reruns-delay 1 --only-rerun TimeoutError

Maven Surefire has the property surefire.rerunFailingTestsCount; a test that passes on a rerun is counted in the summary as a flake, and its failed attempts are written to the XML report as flakyFailure elements.

Retries reduce the number of blocked merges, but they have costs:

a real intermittent defect, such as a race condition in the product, passes on a retry and reaches production; according to Luo et al., a quarter of the fixes of flaky tests changed the code under test;
the suite becomes slower, because every flaky failure is executed several times;
detection is delayed: Google describes a rule that reports a failure only after three consecutive failures, which turns a 15-minute test into 45 minutes of waiting;
without reporting, flakiness accumulates unnoticed.

For this reason retries are combined with reporting: the status "flaky" is recorded in the history, counted and treated as a reason for a fix. An intermittent failure that turns out to be a product defect is reported with everything needed to reproduce it; see the article Defect report. How retries are configured in a pipeline is described in the article Automated tests in CI.

Fixing flaky tests

Waiting for a condition instead of a fixed pause

A fixed pause either wastes time or is too short. The test below waits two seconds after saving and then counts rows once:

import { test, expect } from '@playwright/test';

test('order appears in the list after saving', async ({ page }) => {
  await page.goto('https://example.com/orders/new');
  await page.getByLabel('Customer').fill('Anna');
  await page.getByRole('button', { name: 'Save' }).click();
  await page.waitForTimeout(2000);
  expect(await page.getByRole('row').count()).toBe(2);
});

The Playwright documentation marks page.waitForTimeout as discouraged and states that tests that wait for time are inherently flaky. The fixed version uses an assertion that retries until the condition holds or the timeout expires:

import { test, expect } from '@playwright/test';

test('order appears in the list after saving', async ({ page }) => {
  await page.goto('https://example.com/orders/new');
  await page.getByLabel('Customer').fill('Anna');
  await page.getByRole('button', { name: 'Save' }).click();
  await expect(page.getByRole('row')).toHaveCount(2);
});

Playwright also waits before actions: before a click it checks that the element is visible, stable, receives events and is enabled. The CTAL-TAE syllabus ranks three kinds of waits: a hard-coded wait is the worst, polling for a state change with a timeout is better, and subscribing to an event of the system under test is the most reliable. For a state outside the page, Playwright offers expect.poll:

import { test, expect } from '@playwright/test';

test('payment is confirmed by the back end', async ({ request }) => {
  await expect.poll(async () => {
    const response = await request.get('https://example.com/api/orders/1042');
    return (await response.json()).status;
  }, { timeout: 10000 }).toBe('PAID');
});

Own data for every test

A test creates the data it needs and removes it afterwards. Unique values prevent collisions between parallel tests:

import uuid

import pytest


@pytest.fixture
def customer(api):
    created = api.post("/customers", json={"name": f"test-{uuid.uuid4()}"}).json()
    yield created
    api.delete(f"/customers/{created['id']}")


def test_rename_customer(api, customer):
    api.patch(f"/customers/{customer['id']}", json={"name": "Renamed"})
    assert api.get(f"/customers/{customer['id']}").json()["name"] == "Renamed"

Here api is a fixture of the project with an HTTP client. The test does not depend on records left by other tests, and its own record does not affect them.

Controlled time and external services

Time is fixed by the test rather than taken from the agent. In Playwright, page.clock.setFixedTime sets the time that the page sees, and the option timezoneId sets the time zone of the browser context:

import { test, expect } from '@playwright/test';

test.use({ timezoneId: 'Europe/Berlin' });

test('shows the end of the billing period', async ({ page }) => {
  await page.clock.setFixedTime(new Date('2026-01-31T23:30:00+01:00'));
  await page.goto('https://example.com/billing');
  await expect(page.getByText('Billing period ends today')).toBeVisible();
});

Unstable external services are replaced with test doubles. In Playwright, page.route intercepts requests of the page and returns a prepared response, so the test no longer depends on the availability of the real API.

Flaky test, intermittent defect and broken test

The table shows three situations that look similar in a report.

	Flaky test	Intermittent defect	Broken test
Where the cause is	In the test or its environment	In the product	In the test
Outcome on the same code	Varies	Varies	Always fails
Rerun	Usually passes	Sometimes passes	Fails again
What to fix	Test, data, environment	Product code	Test
Record	Task for the test, quarantine	Defect report	Task for the test

A flaky test and an intermittent defect cannot be distinguished by the outcome alone. Both produce passes and failures on the same code. The difference is found by analysis: logs, traces, reproduction under load. A retry that turns a failure into a pass removes the signal for both cases equally.

A broken test fails deterministically, for example after a change of the interface that the test was not updated for. It is not flaky: its failure is reproducible and points to a concrete change.

Terms

Term	Meaning
Flaky test	Automated test that both passes and fails on the same code without changes to the product, the test or the configuration
Test order dependency	Dependency of the outcome of a test on other tests that run before it and leave shared state
Quarantine	Moving a flaky test out of the suites that gate a merge or a release while it still runs separately until it is fixed
Retry	Automatic rerun of a failed test in the same job; a test that passes on a retry is reported as flaky
Intermittent defect	Defect of the product that causes a failure only in some executions on the same code