[{"data":1,"prerenderedAt":82},["ShallowReactive",2],{"wiki-test-automation-flaky-tests-en":3},{"meta":4,"html":8,"headings":9,"words":81},{"title":5,"description":6,"published":7,"updated":7},"Flaky tests","Flaky tests: tests that pass and fail on the same code. Causes, research data, detection by run history, quarantine, retries and their risks, fixes.","2026-11-03","\u003Cp>A \u003Cstrong>flaky test\u003C/strong> is an automated test that both passes and fails on the same code: the product, the test and the configuration do not change, but the outcome does. A failure of such a test does not show whether the product has a \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"defect\">defect\u003C/span>, and a pass does not show that it has none. Flaky tests are also called non-deterministic or intermittent tests.\u003C/p>\n\u003Cp>The term comes from engineering practice. In 2011 Martin Fowler described the problem in the article "Eradicating Non-Determinism in Tests". In 2016 the Google Testing Blog published "Flaky Tests at Google and How We Mitigate Them" by John Micco, which defines a flaky result as a test showing both a passing and a failing result with the same code. The first large empirical study of the causes, "An Empirical Analysis of Flaky Tests" by Luo, Hariri, Eloussi and Marinov, was presented at the FSE conference in 2014. The ISTQB Advanced Level Test Automation Engineering syllabus (version 2.0, 2024) does not use the word "flaky": it speaks of \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-case\">test cases\u003C/span> that do not give a reliable result, for example because of race conditions, and of intermittent failures. Test runners use "flaky" as a status: Playwright and Maven Surefire mark a test that failed and then passed on a retry as flaky.\u003C/p>\n\u003Ch2 id=\"the-problem-flaky-tests-cause\" class=\"wiki-heading\">The problem flaky tests cause\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"the-problem-flaky-tests-cause\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>An automated test is useful when its failure means a problem in the product. A flaky test breaks this link. After several false failures the team starts to rerun the \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"ci-pipeline\">pipeline\u003C/span> instead of reading the failure. Fowler notes the consequence: once developers learn to ignore the failures of one test, they also ignore the real failures of the others.\u003C/p>\n\u003Cp>The scale of the problem in a large code base was published by Google in 2016:\u003C/p>\n\u003Cul>\n\u003Cli>about 1.5 percent of all \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-run\">test runs\u003C/span> reported a flaky result;\u003C/li>\n\u003Cli>almost 16 percent of tests had some level of flakiness;\u003C/li>\n\u003Cli>about 84 percent of the observed transitions from pass to fail in post-submit testing involved a flaky test.\u003C/li>\n\u003C/ul>\n\u003Cp>For a project with 1000 tests and a flake rate of 1.5 percent, about 15 tests would fail in a run for reasons unrelated to the change. Every such failure blocks a merge, costs time for analysis and lowers trust in the results of the whole suite.\u003C/p>\n\u003Ch2 id=\"causes-of-flakiness\" class=\"wiki-heading\">Causes of flakiness\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"causes-of-flakiness\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>A test is flaky when its outcome depends on something that the test does not control. The usual sources of such dependencies are described below.\u003C/p>\n\u003Ch3 id=\"asynchronous-waits-and-timing\" class=\"wiki-heading\">Asynchronous waits and timing\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"asynchronous-waits-and-timing\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>The test performs an action and checks the result before the asynchronous operation has finished: a request to the server, a page render, a message in a queue. A fixed pause in the test hides the problem as long as the operation is faster than the pause. On a loaded CI agent the operation takes longer, and the test fails. The ISTQB CTAL-TAE syllabus names hard-coded waits as a root cause of many test failures.\u003C/p>\n\u003Ch3 id=\"test-order-dependency-and-shared-state\" class=\"wiki-heading\">Test order dependency and shared state\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"test-order-dependency-and-shared-state\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>One test leaves state that another test relies on: a record in the database, a file, a value in a cache, a static variable. The second test passes when it runs after the first one and fails when the order changes, for example when tests are distributed between parallel workers. Fowler recommends keeping tests isolated, so that the execution of one test does not affect any other.\u003C/p>\n\u003Ch3 id=\"test-data\" class=\"wiki-heading\">Test data\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"test-data\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>Tests that share accounts or records interfere with each other. Two tests that rename the same customer in parallel see each other's changes. A test that expects an empty list fails when another test has added an element. Data that expires, such as a token, a trial period or a reserved slot, makes a test depend on the date of the run.\u003C/p>\n\u003Ch3 id=\"environment-and-external-services\" class=\"wiki-heading\">Environment and external services\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"environment-and-external-services\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>The test depends on a network, a third-party API, a mail server, a shared \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-environment\">test environment\u003C/span>. A timeout or a temporary error of such a service fails the test without any change in the product. Fowler suggests replacing unstable remote services with test doubles, stand-ins that behave like the real service. Differences between agents also play a role: the number of cores, the available memory, the screen resolution, installed fonts.\u003C/p>\n\u003Ch3 id=\"concurrency\" class=\"wiki-heading\">Concurrency\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"concurrency\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>The product or the test itself has several threads or processes, and the result depends on the order in which they run. A race condition shows up only with some interleavings of operations. Such defects are often real defects of the product, not only of the test.\u003C/p>\n\u003Ch3 id=\"randomness-time-and-time-zones\" class=\"wiki-heading\">Randomness, time and time zones\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"randomness-time-and-time-zones\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>Random input data, random identifiers and iteration over unordered collections give a different execution on every run. Tests that read the system clock fail at the end of the month, at midnight, on a change of daylight saving time or on an agent in another time zone. Fowler recommends always wrapping the system clock, so that a test can substitute it.\u003C/p>\n\u003Ch3 id=\"resource-leaks\" class=\"wiki-heading\">Resource leaks\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"resource-leaks\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>A test does not release a connection, a file handle, a port, a browser process or memory. Subsequent tests fail when the resource runs out, and the failure appears in a test that has nothing to do with the leak. Fowler describes a resource pool of size one as a way to make the first test after a leak fail and point to the cause.\u003C/p>\n\u003Ch2 id=\"empirical-data\" class=\"wiki-heading\">Empirical data\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"empirical-data\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>Luo, Hariri, Eloussi and Marinov studied commits in the Apache Software Foundation repositories whose messages mention intermittent or flaky tests. They found 486 commits that likely fix distinct flaky tests in at least 51 projects and analyzed 201 of them in detail. For 161 commits the root cause could be classified. The distribution of root causes:\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-3\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Root cause\u003C/th>\n\u003Cth>Commits\u003C/th>\n\u003Cth>Share of 161\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Asynchronous wait\u003C/td>\n\u003Ctd>74\u003C/td>\n\u003Ctd>45%\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Concurrency\u003C/td>\n\u003Ctd>32\u003C/td>\n\u003Ctd>20%\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Test order dependency\u003C/td>\n\u003Ctd>19\u003C/td>\n\u003Ctd>12%\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Resource leak\u003C/td>\n\u003Ctd>11\u003C/td>\n\u003Ctd>7%\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Network\u003C/td>\n\u003Ctd>10\u003C/td>\n\u003Ctd>6%\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Time\u003C/td>\n\u003Ctd>5\u003C/td>\n\u003Ctd>3%\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Input and output\u003C/td>\n\u003Ctd>4\u003C/td>\n\u003Ctd>2%\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Randomness\u003C/td>\n\u003Ctd>4\u003C/td>\n\u003Ctd>2%\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Floating point operations\u003C/td>\n\u003Ctd>3\u003C/td>\n\u003Ctd>2%\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Unordered collections\u003C/td>\n\u003Ctd>1\u003C/td>\n\u003Ctd>1%\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>The study also reports how the tests were fixed. 42 of the 74 asynchronous wait tests were fixed with an \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"explicit-wait\">explicit wait\u003C/span> for a condition; 20 were "fixed" by adding or extending a sleep, which only reduced the chance of failure. All fixes of order-dependent tests removed the flakiness, most often by setting up or cleaning shared state. In 38 of 161 cases (24 percent) the fix also changed the code under test, and in most of these cases it fixed a real defect. 126 of the 161 tests were flaky from the moment they were written.\u003C/p>\n\u003Ch2 id=\"detecting-flaky-tests\" class=\"wiki-heading\">Detecting flaky tests\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"detecting-flaky-tests\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Ch3 id=\"reruns-on-the-same-commit\" class=\"wiki-heading\">Reruns on the same commit\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"reruns-on-the-same-commit\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>The direct way to show flakiness is to run a test several times on the same commit without any changes. If the outcomes differ, the test is flaky by definition. Playwright runs every test several times with the option \u003Ccode>--repeat-each\u003C/code>, and a failed job can be repeated with \u003Ccode>--last-failed\u003C/code>, which runs only the tests that failed. A rerun confirms flakiness but does not find the cause.\u003C/p>\n\u003Ch3 id=\"history-of-runs\" class=\"wiki-heading\">History of runs\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"history-of-runs\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>Every run of an automated suite produces a result for each test. When results are kept as a history, flakiness becomes visible as flips between passed and failed without a relevant change:\u003C/p>\n\u003Cdiv class=\"wiki-code wiki-code--plain\">\u003Cpre>\u003Ccode>Test "checkout > pay with saved card", branch main\n\ncommit   run    result\na41f2c   #812   passed\na41f2c   #813   failed    rerun of the same commit\nb7d903   #814   passed\nc1e5aa   #815   passed\nc1e5aa   #816   failed    rerun of the same commit\nd90b4e   #817   passed\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>Two failures on commits that also passed are a sign of a flaky test, not of a regression. A test that fails on every run starting from one commit points to a change. Results of runs and their history in a \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-management-system\">test management system\u003C/span> are described in the article \u003Ca href=\"/en/wiki/test-management/test-run\" class=\"wiki-link wiki-link--article\">Test run\u003C/a>. Metrics derived from such history include the share of runs with a flip and the number of flips per test over a period.\u003C/p>\n\u003Ch3 id=\"dedicated-jobs\" class=\"wiki-heading\">Dedicated jobs\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"dedicated-jobs\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>Some causes appear only under particular conditions. Teams run separate jobs to expose them:\u003C/p>\n\u003Cul>\n\u003Cli>a nightly job that repeats every test many times;\u003C/li>\n\u003Cli>a job that runs tests in a different or random order, to find order dependencies;\u003C/li>\n\u003Cli>a job that runs each test in isolation, to separate the test from its neighbors;\u003C/li>\n\u003Cli>a job on an agent with limited resources, where timing problems appear more often.\u003C/li>\n\u003C/ul>\n\u003Cp>The CTAL-TAE syllabus recommends executing a few isolated tests when analyzing an unexpected result and notes that monitoring system resources can give clues to the root cause of intermittent failures.\u003C/p>\n\u003Ch2 id=\"quarantine\" class=\"wiki-heading\">Quarantine\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"quarantine\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>Quarantine is the practice of moving a flaky test out of the suites whose results gate a merge or a release, while the test continues to run separately. Fowler describes it as keeping non-deterministic tests in a different suite from the healthy tests. The Google post describes a tool that quarantines tests with too high flakiness and files a bug for each of them; the same post warns that this can hide real defects.\u003C/p>\n\u003Cp>A quarantine policy usually defines:\u003C/p>\n\u003Cul>\n\u003Cli>\u003Cstrong>entry criterion\u003C/strong>: for example, two failures and passes on the same commit within a week;\u003C/li>\n\u003Cli>\u003Cstrong>owner\u003C/strong>: the team responsible for the test, and a defect or task for the fix;\u003C/li>\n\u003Cli>\u003Cstrong>limit\u003C/strong>: Fowler suggests limiting the number of tests in quarantine (for example, eight) or the time a test may stay there (for example, a week);\u003C/li>\n\u003Cli>\u003Cstrong>exit criterion\u003C/strong>: the fix is merged and the test passes a series of repeated runs;\u003C/li>\n\u003Cli>\u003Cstrong>visibility\u003C/strong>: quarantined tests still run and their results are reported, so that a real regression in them is noticed.\u003C/li>\n\u003C/ul>\n\u003Cp>A test that stays in quarantine without an owner no longer checks anything. Removing it is then an explicit decision rather than a side effect.\u003C/p>\n\u003Ch2 id=\"retries-and-their-risks\" class=\"wiki-heading\">Retries and their risks\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"retries-and-their-risks\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>A retry reruns a failed test automatically in the same job. Test runners support it directly. In Playwright the option \u003Ccode>retries\u003C/code> sets the maximum number of retries, and a test that failed first and passed on a retry gets the status "flaky":\u003C/p>\n\u003Cdiv class=\"wiki-code\" data-language=\"typescript\">\u003Cdiv class=\"wiki-code__header\">\u003Cspan class=\"wiki-code__language\">typescript\u003C/span>\u003Cbutton type=\"button\" class=\"wiki-code__copy\" data-wiki-copy data-copied-label=\"Copied\">\u003Csvg class=\"wiki-code__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M216,28H88A12,12,0,0,0,76,40V76H40A12,12,0,0,0,28,88V216a12,12,0,0,0,12,12H168a12,12,0,0,0,12-12V180h36a12,12,0,0,0,12-12V40A12,12,0,0,0,216,28ZM156,204H52V100H156Zm48-48H180V88a12,12,0,0,0-12-12H100V52H204Z\"/>\u003C/svg>\u003Csvg class=\"wiki-code__icon wiki-code__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003Cspan class=\"wiki-code__label\">Copy\u003C/span>\u003Cspan class=\"wiki-code__label wiki-code__label--done\">Copied\u003C/span>\u003C/button>\u003C/div>\u003Cpre>\u003Ccode>import { defineConfig } from '@playwright/test';\n\nexport default defineConfig({\n  retries: process.env.CI ? 2 : 0,\n  use: {\n    trace: 'on-first-retry',\n  },\n});\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>The trace is recorded on the first retry, so a flaky failure leaves data for analysis. Since version 1.52 the option \u003Ccode>failOnFlakyTests\u003C/code> (on the command line \u003Ccode>--fail-on-flaky-tests\u003C/code>) makes the run fail if any test is flaky. In pytest the plugin \u003Ccode>pytest-rerunfailures\u003C/code> adds the options \u003Ccode>--reruns\u003C/code>, \u003Ccode>--reruns-delay\u003C/code> and \u003Ccode>--only-rerun\u003C/code>, the last of which retries only failures that match a pattern:\u003C/p>\n\u003Cdiv class=\"wiki-code\" data-language=\"bash\">\u003Cdiv class=\"wiki-code__header\">\u003Cspan class=\"wiki-code__language\">Bash\u003C/span>\u003Cbutton type=\"button\" class=\"wiki-code__copy\" data-wiki-copy data-copied-label=\"Copied\">\u003Csvg class=\"wiki-code__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M216,28H88A12,12,0,0,0,76,40V76H40A12,12,0,0,0,28,88V216a12,12,0,0,0,12,12H168a12,12,0,0,0,12-12V180h36a12,12,0,0,0,12-12V40A12,12,0,0,0,216,28ZM156,204H52V100H156Zm48-48H180V88a12,12,0,0,0-12-12H100V52H204Z\"/>\u003C/svg>\u003Csvg class=\"wiki-code__icon wiki-code__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003Cspan class=\"wiki-code__label\">Copy\u003C/span>\u003Cspan class=\"wiki-code__label wiki-code__label--done\">Copied\u003C/span>\u003C/button>\u003C/div>\u003Cpre>\u003Ccode>pytest --reruns 2 --reruns-delay 1 --only-rerun TimeoutError\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>Maven Surefire has the property \u003Ccode>surefire.rerunFailingTestsCount\u003C/code>; a test that passes on a rerun is counted in the summary as a flake, and its failed attempts are written to the XML report as \u003Ccode>flakyFailure\u003C/code> elements.\u003C/p>\n\u003Cp>Retries reduce the number of blocked merges, but they have costs:\u003C/p>\n\u003Cul>\n\u003Cli>a real intermittent defect, such as a race condition in the product, passes on a retry and reaches production; according to Luo et al., a quarter of the fixes of flaky tests changed the code under test;\u003C/li>\n\u003Cli>the suite becomes slower, because every flaky failure is executed several times;\u003C/li>\n\u003Cli>detection is delayed: Google describes a rule that reports a failure only after three consecutive failures, which turns a 15-minute test into 45 minutes of waiting;\u003C/li>\n\u003Cli>without reporting, flakiness accumulates unnoticed.\u003C/li>\n\u003C/ul>\n\u003Cp>For this reason retries are combined with reporting: the status "flaky" is recorded in the history, counted and treated as a reason for a fix. An intermittent failure that turns out to be a product defect is reported with everything needed to reproduce it; see the article \u003Ca href=\"/en/wiki/test-management/defect-report\" class=\"wiki-link wiki-link--article\">Defect report\u003C/a>. How retries are configured in a pipeline is described in the article \u003Ca href=\"/en/wiki/test-automation/automated-tests-in-ci\" class=\"wiki-link wiki-link--article\">Automated tests in CI\u003C/a>.\u003C/p>\n\u003Ch2 id=\"fixing-flaky-tests\" class=\"wiki-heading\">Fixing flaky tests\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"fixing-flaky-tests\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Ch3 id=\"waiting-for-a-condition-instead-of-a-fixed-pause\" class=\"wiki-heading\">Waiting for a condition instead of a fixed pause\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"waiting-for-a-condition-instead-of-a-fixed-pause\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>A fixed pause either wastes time or is too short. The test below waits two seconds after saving and then counts rows once:\u003C/p>\n\u003Cdiv class=\"wiki-code\" data-language=\"typescript\">\u003Cdiv class=\"wiki-code__header\">\u003Cspan class=\"wiki-code__language\">typescript\u003C/span>\u003Cbutton type=\"button\" class=\"wiki-code__copy\" data-wiki-copy data-copied-label=\"Copied\">\u003Csvg class=\"wiki-code__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M216,28H88A12,12,0,0,0,76,40V76H40A12,12,0,0,0,28,88V216a12,12,0,0,0,12,12H168a12,12,0,0,0,12-12V180h36a12,12,0,0,0,12-12V40A12,12,0,0,0,216,28ZM156,204H52V100H156Zm48-48H180V88a12,12,0,0,0-12-12H100V52H204Z\"/>\u003C/svg>\u003Csvg class=\"wiki-code__icon wiki-code__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003Cspan class=\"wiki-code__label\">Copy\u003C/span>\u003Cspan class=\"wiki-code__label wiki-code__label--done\">Copied\u003C/span>\u003C/button>\u003C/div>\u003Cpre>\u003Ccode>import { test, expect } from '@playwright/test';\n\ntest('order appears in the list after saving', async ({ page }) => {\n  await page.goto('https://example.com/orders/new');\n  await page.getByLabel('Customer').fill('Anna');\n  await page.getByRole('button', { name: 'Save' }).click();\n  await page.waitForTimeout(2000);\n  expect(await page.getByRole('row').count()).toBe(2);\n});\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>The Playwright documentation marks \u003Ccode>page.waitForTimeout\u003C/code> as discouraged and states that tests that wait for time are inherently flaky. The fixed version uses an assertion that retries until the condition holds or the timeout expires:\u003C/p>\n\u003Cdiv class=\"wiki-code\" data-language=\"typescript\">\u003Cdiv class=\"wiki-code__header\">\u003Cspan class=\"wiki-code__language\">typescript\u003C/span>\u003Cbutton type=\"button\" class=\"wiki-code__copy\" data-wiki-copy data-copied-label=\"Copied\">\u003Csvg class=\"wiki-code__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M216,28H88A12,12,0,0,0,76,40V76H40A12,12,0,0,0,28,88V216a12,12,0,0,0,12,12H168a12,12,0,0,0,12-12V180h36a12,12,0,0,0,12-12V40A12,12,0,0,0,216,28ZM156,204H52V100H156Zm48-48H180V88a12,12,0,0,0-12-12H100V52H204Z\"/>\u003C/svg>\u003Csvg class=\"wiki-code__icon wiki-code__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003Cspan class=\"wiki-code__label\">Copy\u003C/span>\u003Cspan class=\"wiki-code__label wiki-code__label--done\">Copied\u003C/span>\u003C/button>\u003C/div>\u003Cpre>\u003Ccode>import { test, expect } from '@playwright/test';\n\ntest('order appears in the list after saving', async ({ page }) => {\n  await page.goto('https://example.com/orders/new');\n  await page.getByLabel('Customer').fill('Anna');\n  await page.getByRole('button', { name: 'Save' }).click();\n  await expect(page.getByRole('row')).toHaveCount(2);\n});\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>Playwright also waits before actions: before a click it checks that the element is visible, stable, receives events and is enabled. The CTAL-TAE syllabus ranks three kinds of waits: a hard-coded wait is the worst, polling for a state change with a timeout is better, and subscribing to an event of the system under test is the most reliable. For a state outside the page, Playwright offers \u003Ccode>expect.poll\u003C/code>:\u003C/p>\n\u003Cdiv class=\"wiki-code\" data-language=\"typescript\">\u003Cdiv class=\"wiki-code__header\">\u003Cspan class=\"wiki-code__language\">typescript\u003C/span>\u003Cbutton type=\"button\" class=\"wiki-code__copy\" data-wiki-copy data-copied-label=\"Copied\">\u003Csvg class=\"wiki-code__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M216,28H88A12,12,0,0,0,76,40V76H40A12,12,0,0,0,28,88V216a12,12,0,0,0,12,12H168a12,12,0,0,0,12-12V180h36a12,12,0,0,0,12-12V40A12,12,0,0,0,216,28ZM156,204H52V100H156Zm48-48H180V88a12,12,0,0,0-12-12H100V52H204Z\"/>\u003C/svg>\u003Csvg class=\"wiki-code__icon wiki-code__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003Cspan class=\"wiki-code__label\">Copy\u003C/span>\u003Cspan class=\"wiki-code__label wiki-code__label--done\">Copied\u003C/span>\u003C/button>\u003C/div>\u003Cpre>\u003Ccode>import { test, expect } from '@playwright/test';\n\ntest('payment is confirmed by the back end', async ({ request }) => {\n  await expect.poll(async () => {\n    const response = await request.get('https://example.com/api/orders/1042');\n    return (await response.json()).status;\n  }, { timeout: 10000 }).toBe('PAID');\n});\u003C/code>\u003C/pre>\u003C/div>\n\u003Ch3 id=\"own-data-for-every-test\" class=\"wiki-heading\">Own data for every test\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"own-data-for-every-test\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>A test creates the data it needs and removes it afterwards. Unique values prevent collisions between parallel tests:\u003C/p>\n\u003Cdiv class=\"wiki-code\" data-language=\"python\">\u003Cdiv class=\"wiki-code__header\">\u003Cspan class=\"wiki-code__language\">python\u003C/span>\u003Cbutton type=\"button\" class=\"wiki-code__copy\" data-wiki-copy data-copied-label=\"Copied\">\u003Csvg class=\"wiki-code__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M216,28H88A12,12,0,0,0,76,40V76H40A12,12,0,0,0,28,88V216a12,12,0,0,0,12,12H168a12,12,0,0,0,12-12V180h36a12,12,0,0,0,12-12V40A12,12,0,0,0,216,28ZM156,204H52V100H156Zm48-48H180V88a12,12,0,0,0-12-12H100V52H204Z\"/>\u003C/svg>\u003Csvg class=\"wiki-code__icon wiki-code__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003Cspan class=\"wiki-code__label\">Copy\u003C/span>\u003Cspan class=\"wiki-code__label wiki-code__label--done\">Copied\u003C/span>\u003C/button>\u003C/div>\u003Cpre>\u003Ccode>import uuid\n\nimport pytest\n\n\n@pytest.fixture\ndef customer(api):\n    created = api.post("/customers", json={"name": f"test-{uuid.uuid4()}"}).json()\n    yield created\n    api.delete(f"/customers/{created['id']}")\n\n\ndef test_rename_customer(api, customer):\n    api.patch(f"/customers/{customer['id']}", json={"name": "Renamed"})\n    assert api.get(f"/customers/{customer['id']}").json()["name"] == "Renamed"\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>Here \u003Ccode>api\u003C/code> is a fixture of the project with an HTTP client. The test does not depend on records left by other tests, and its own record does not affect them.\u003C/p>\n\u003Ch3 id=\"controlled-time-and-external-services\" class=\"wiki-heading\">Controlled time and external services\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"controlled-time-and-external-services\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>Time is fixed by the test rather than taken from the agent. In Playwright, \u003Ccode>page.clock.setFixedTime\u003C/code> sets the time that the page sees, and the option \u003Ccode>timezoneId\u003C/code> sets the time zone of the browser context:\u003C/p>\n\u003Cdiv class=\"wiki-code\" data-language=\"typescript\">\u003Cdiv class=\"wiki-code__header\">\u003Cspan class=\"wiki-code__language\">typescript\u003C/span>\u003Cbutton type=\"button\" class=\"wiki-code__copy\" data-wiki-copy data-copied-label=\"Copied\">\u003Csvg class=\"wiki-code__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M216,28H88A12,12,0,0,0,76,40V76H40A12,12,0,0,0,28,88V216a12,12,0,0,0,12,12H168a12,12,0,0,0,12-12V180h36a12,12,0,0,0,12-12V40A12,12,0,0,0,216,28ZM156,204H52V100H156Zm48-48H180V88a12,12,0,0,0-12-12H100V52H204Z\"/>\u003C/svg>\u003Csvg class=\"wiki-code__icon wiki-code__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003Cspan class=\"wiki-code__label\">Copy\u003C/span>\u003Cspan class=\"wiki-code__label wiki-code__label--done\">Copied\u003C/span>\u003C/button>\u003C/div>\u003Cpre>\u003Ccode>import { test, expect } from '@playwright/test';\n\ntest.use({ timezoneId: 'Europe/Berlin' });\n\ntest('shows the end of the billing period', async ({ page }) => {\n  await page.clock.setFixedTime(new Date('2026-01-31T23:30:00+01:00'));\n  await page.goto('https://example.com/billing');\n  await expect(page.getByText('Billing period ends today')).toBeVisible();\n});\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>Unstable external services are replaced with test doubles. In Playwright, \u003Ccode>page.route\u003C/code> intercepts requests of the page and returns a prepared response, so the test no longer depends on the availability of the real API.\u003C/p>\n\u003Ch2 id=\"flaky-test-intermittent-defect-and-broken-test\" class=\"wiki-heading\">Flaky test, intermittent defect and broken test\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"flaky-test-intermittent-defect-and-broken-test\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The table shows three situations that look similar in a report.\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-4\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>\u003C/th>\n\u003Cth>Flaky test\u003C/th>\n\u003Cth>Intermittent defect\u003C/th>\n\u003Cth>Broken test\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Where the cause is\u003C/td>\n\u003Ctd>In the test or its environment\u003C/td>\n\u003Ctd>In the product\u003C/td>\n\u003Ctd>In the test\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Outcome on the same code\u003C/td>\n\u003Ctd>Varies\u003C/td>\n\u003Ctd>Varies\u003C/td>\n\u003Ctd>Always fails\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Rerun\u003C/td>\n\u003Ctd>Usually passes\u003C/td>\n\u003Ctd>Sometimes passes\u003C/td>\n\u003Ctd>Fails again\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>What to fix\u003C/td>\n\u003Ctd>Test, data, environment\u003C/td>\n\u003Ctd>Product code\u003C/td>\n\u003Ctd>Test\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Record\u003C/td>\n\u003Ctd>Task for the test, quarantine\u003C/td>\n\u003Ctd>Defect report\u003C/td>\n\u003Ctd>Task for the test\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>A flaky test and an intermittent defect cannot be distinguished by the outcome alone. Both produce passes and failures on the same code. The difference is found by analysis: logs, traces, reproduction under load. A retry that turns a failure into a pass removes the signal for both cases equally.\u003C/p>\n\u003Cp>A broken test fails deterministically, for example after a change of the interface that the test was not updated for. It is not flaky: its failure is reproducible and points to a concrete change.\u003C/p>\n\u003Ch2 id=\"terms\" class=\"wiki-heading\">Terms\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"terms\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cdiv class=\"wiki-table wiki-table--cols-2\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Term\u003C/th>\n\u003Cth>Meaning\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Flaky test\u003C/td>\n\u003Ctd>Automated test that both passes and fails on the same code without changes to the product, the test or the configuration\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Test order dependency\u003C/td>\n\u003Ctd>Dependency of the outcome of a test on other tests that run before it and leave shared state\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Quarantine\u003C/td>\n\u003Ctd>Moving a flaky test out of the suites that gate a merge or a release while it still runs separately until it is fixed\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Retry\u003C/td>\n\u003Ctd>Automatic rerun of a failed test in the same job; a test that passes on a retry is reported as flaky\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Intermittent defect\u003C/td>\n\u003Ctd>Defect of the product that causes a failure only in some executions on the same code\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Ch2 id=\"sources\" class=\"wiki-heading\">Sources\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"sources\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cul>\n\u003Cli>\u003Ca href=\"https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Flaky Tests at Google and How We Mitigate Them, John Micco, Google Testing Blog, 27 May 2016\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://martinfowler.com/articles/nonDeterminism.html\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Eradicating Non-Determinism in Tests, Martin Fowler, 14 April 2011\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://huang.isis.vanderbilt.edu/cs8395/paper/flakytest.pdf\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Luo, Hariri, Eloussi, Marinov. An Empirical Analysis of Flaky Tests, FSE 2014\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://www.istqb.org/wp-content/uploads/2024/11/ISTQB_CTAL-TAE_Syllabus_v2.0.pdf\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">ISTQB Certified Tester Advanced Level Test Automation Engineering Syllabus v2.0, ISTQB, 2024\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://playwright.dev/docs/test-retries\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Retries, Playwright documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://playwright.dev/docs/actionability\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Auto-waiting, Playwright documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://playwright.dev/docs/test-assertions\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Assertions, Playwright documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://playwright.dev/docs/clock\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Clock, Playwright documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://playwright.dev/docs/api/class-testconfig\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">TestConfig, Playwright documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://github.com/pytest-dev/pytest-rerunfailures\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">pytest-rerunfailures, pytest-dev\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Rerun failing tests, Maven Surefire Plugin\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003C/ul>\n",[10,14,17,21,24,27,30,33,36,39,42,45,48,51,54,57,60,63,66,69,72,75,78],{"id":11,"level":12,"text":13},"the-problem-flaky-tests-cause",2,"The problem flaky tests cause",{"id":15,"level":12,"text":16},"causes-of-flakiness","Causes of flakiness",{"id":18,"level":19,"text":20},"asynchronous-waits-and-timing",3,"Asynchronous waits and timing",{"id":22,"level":19,"text":23},"test-order-dependency-and-shared-state","Test order dependency and shared state",{"id":25,"level":19,"text":26},"test-data","Test data",{"id":28,"level":19,"text":29},"environment-and-external-services","Environment and external services",{"id":31,"level":19,"text":32},"concurrency","Concurrency",{"id":34,"level":19,"text":35},"randomness-time-and-time-zones","Randomness, time and time zones",{"id":37,"level":19,"text":38},"resource-leaks","Resource leaks",{"id":40,"level":12,"text":41},"empirical-data","Empirical data",{"id":43,"level":12,"text":44},"detecting-flaky-tests","Detecting flaky tests",{"id":46,"level":19,"text":47},"reruns-on-the-same-commit","Reruns on the same commit",{"id":49,"level":19,"text":50},"history-of-runs","History of runs",{"id":52,"level":19,"text":53},"dedicated-jobs","Dedicated jobs",{"id":55,"level":12,"text":56},"quarantine","Quarantine",{"id":58,"level":12,"text":59},"retries-and-their-risks","Retries and their risks",{"id":61,"level":12,"text":62},"fixing-flaky-tests","Fixing flaky tests",{"id":64,"level":19,"text":65},"waiting-for-a-condition-instead-of-a-fixed-pause","Waiting for a condition instead of a fixed pause",{"id":67,"level":19,"text":68},"own-data-for-every-test","Own data for every test",{"id":70,"level":19,"text":71},"controlled-time-and-external-services","Controlled time and external services",{"id":73,"level":12,"text":74},"flaky-test-intermittent-defect-and-broken-test","Flaky test, intermittent defect and broken test",{"id":76,"level":12,"text":77},"terms","Terms",{"id":79,"level":12,"text":80},"sources","Sources",2461,1793718905866]