[{"data":1,"prerenderedAt":51},["ShallowReactive",2],{"wiki-test-automation-automated-tests-in-ci-en":3},{"meta":4,"html":8,"headings":9,"words":50},{"title":5,"description":6,"published":7,"updated":7},"Automated tests in CI","Automated tests in CI: pipeline triggers, stages by speed, parallelism and sharding, JUnit XML reports, retries, artifacts, results in test runs.","2026-11-03","\u003Cp>\u003Cstrong>Continuous integration\u003C/strong> (CI) is a development practice in which the members of a team merge their changes into a shared mainline at least daily, and every integration is verified by an automated build that includes tests. Automated tests are the part of this build that decides whether a change can be accepted. A CI system starts them on events in the code repository and on a schedule, collects their results and reports them back to the change, the team and other systems.\u003C/p>\n\u003Cp>Continuous integration was developed by Kent Beck as a practice of Extreme Programming in the 1990s. Martin Fowler published the article "Continuous Integration" in 2000 and revised it in January 2024; it lists the practices, among them a self-testing build, a build on every push to the mainline and a fast build. The term itself is sometimes credited to Grady Booch, who, according to Fowler, mentioned it only in passing in a book on object-oriented design. The ISTQB Advanced Level Test Automation Engineering syllabus (version 2.0, 2024) devotes a chapter to the integration of automated tests into CI/CD pipelines: which test levels run at which stage and which of them act as quality gates.\u003C/p>\n\u003Ch2 id=\"the-problem-ci-solves\" class=\"wiki-heading\">The problem CI solves\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"the-problem-ci-solves\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>Without CI, changes are integrated rarely and in large portions. Conflicts and \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"defect\">defects\u003C/span> accumulate, and their cause is hard to find because many changes have been made since the last working state. Tests that are run by hand before a release are run late, on an unknown combination of changes, and sometimes not at all.\u003C/p>\n\u003Cp>CI shortens the distance between a change and its check. Every push or merge request starts the build and the tests, and the author gets the result while the change is still fresh. Fowler formulates the requirement for speed through the Extreme Programming guideline of a ten-minute build. A failed build is fixed immediately, so that the mainline stays in a working state.\u003C/p>\n\u003Cp>The levels of automated tests (unit, integration, end-to-end) and their cost are described in the article \u003Ca href=\"/en/wiki/test-automation/automation-levels-and-test-pyramid\" class=\"wiki-link wiki-link--article\">Test automation levels and the test pyramid\u003C/a>. This article describes how these tests are run in CI and how their results are used.\u003C/p>\n\u003Ch2 id=\"pipeline-triggers\" class=\"wiki-heading\">Pipeline triggers\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"pipeline-triggers\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>A \u003Cstrong>pipeline\u003C/strong> is a sequence of jobs that a CI system executes for one event: building, running tests, publishing artifacts, deploying. A pipeline is started by a trigger. The usual triggers and the tests they run:\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-3\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Trigger\u003C/th>\n\u003Cth>When it fires\u003C/th>\n\u003Cth>Typical set of tests\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Push\u003C/td>\n\u003Ctd>A commit is pushed to a branch\u003C/td>\n\u003Ctd>Fast tests: unit tests, static checks\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Pull or merge request\u003C/td>\n\u003Ctd>A request is opened or updated\u003C/td>\n\u003Ctd>Tests needed to accept the change\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Schedule\u003C/td>\n\u003Ctd>At a set time, for example every night\u003C/td>\n\u003Ctd>Full regression, long and non-functional tests\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Manual\u003C/td>\n\u003Ctd>A person starts the pipeline\u003C/td>\n\u003Ctd>A selected suite on a selected environment\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>In GitHub Actions the triggers are declared in the key \u003Ccode>on\u003C/code> of a workflow:\u003C/p>\n\u003Cdiv class=\"wiki-code\" data-language=\"yaml\">\u003Cdiv class=\"wiki-code__header\">\u003Cspan class=\"wiki-code__language\">yaml\u003C/span>\u003Cbutton type=\"button\" class=\"wiki-code__copy\" data-wiki-copy data-copied-label=\"Copied\">\u003Csvg class=\"wiki-code__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M216,28H88A12,12,0,0,0,76,40V76H40A12,12,0,0,0,28,88V216a12,12,0,0,0,12,12H168a12,12,0,0,0,12-12V180h36a12,12,0,0,0,12-12V40A12,12,0,0,0,216,28ZM156,204H52V100H156Zm48-48H180V88a12,12,0,0,0-12-12H100V52H204Z\"/>\u003C/svg>\u003Csvg class=\"wiki-code__icon wiki-code__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003Cspan class=\"wiki-code__label\">Copy\u003C/span>\u003Cspan class=\"wiki-code__label wiki-code__label--done\">Copied\u003C/span>\u003C/button>\u003C/div>\u003Cpre>\u003Ccode>name: Tests\n\non:\n  push:\n    branches: [ main ]\n  pull_request:\n    branches: [ main ]\n  schedule:\n    - cron: '0 2 * * *'\n  workflow_dispatch:\n    inputs:\n      suite:\n        description: 'Test suite to run'\n        required: true\n        default: 'smoke'\n        type: choice\n        options:\n        - smoke\n        - regression\n\njobs:\n  unit:\n    runs-on: ubuntu-latest\n    steps:\n    - uses: actions/checkout@v7\n    - uses: actions/setup-node@v7\n      with:\n        node-version: lts/*\n    - run: npm ci\n    - run: npm test\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>The GitHub Actions documentation defines the details of each trigger. By default, \u003Ccode>pull_request\u003C/code> fires for the activity types \u003Ccode>opened\u003C/code>, \u003Ccode>synchronize\u003C/code> and \u003Ccode>reopened\u003C/code>, and the workflow runs on the merge commit of the request (\u003Ccode>refs/pull/<number>/merge\u003C/code>), so the tests check the result of the merge rather than the branch alone. A \u003Ccode>schedule\u003C/code> uses POSIX cron syntax, runs in UTC by default on the latest commit of the default branch, cannot run more often than every five minutes and can be delayed when the load is high. In public repositories scheduled workflows are disabled after 60 days without activity. \u003Ccode>workflow_dispatch\u003C/code> works only if the workflow file is on the default branch; its inputs can be of the types \u003Ccode>boolean\u003C/code>, \u003Ccode>choice\u003C/code>, \u003Ccode>number\u003C/code>, \u003Ccode>environment\u003C/code> and \u003Ccode>string\u003C/code>.\u003C/p>\n\u003Ch2 id=\"stages-by-speed\" class=\"wiki-heading\">Stages by speed\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"stages-by-speed\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>Tests in a pipeline are ordered from fast to slow. A fast stage fails early and does not spend resources on slow stages for a change that is already broken. Fowler describes this as a deployment pipeline: the commit build runs on every push and must be fast, and secondary builds run slower tests, for example against a real database, without blocking every commit.\u003C/p>\n\u003Cp>The CTAL-TAE syllabus distributes test levels between stages:\u003C/p>\n\u003Cul>\n\u003Cli>component (unit) tests run in the build phase and act as quality gates of the pipeline; component \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"integration-test\">integration tests\u003C/span> run alongside them;\u003C/li>\n\u003Cli>system integration tests are often quality gates of a continuous delivery pipeline;\u003C/li>\n\u003Cli>system tests can run in a continuous deployment pipeline as the last quality gate before the product is delivered;\u003C/li>\n\u003Cli>system and acceptance tests run either in the deployment phase, where a failure can roll back the deployment, or in a separate pipeline started by a successful deployment.\u003C/li>\n\u003C/ul>\n\u003Cp>A typical order, with illustrative durations:\u003C/p>\n\u003Cdiv class=\"wiki-code wiki-code--plain\">\u003Cpre>\u003Ccode>push / pull request\n |\n +-- build, static checks               1-2 min\n +-- unit tests                         2-5 min    quality gate\n +-- integration tests                  5-10 min   quality gate\n +-- deploy to test environment\n +-- end-to-end tests, 4 shards         10-15 min  quality gate\n +-- report: JUnit XML, artifacts\nnightly schedule\n +-- full regression, performance tests\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>In GitHub Actions the order of jobs is set with \u003Ccode>needs\u003C/code>: a job starts only after the jobs it needs have completed successfully, and if one of them fails or is skipped, the dependent jobs are skipped.\u003C/p>\n\u003Ch2 id=\"parallelism-and-sharding\" class=\"wiki-heading\">Parallelism and sharding\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"parallelism-and-sharding\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>Parallelism inside one machine and distribution between machines are two different levels. Playwright runs test files in parallel worker processes; the number of workers defaults to half of the logical CPU cores. Its CI guide recommends one worker on CI agents for stability and reproducibility, and sharding for wider parallelization.\u003C/p>\n\u003Cp>\u003Cstrong>Sharding\u003C/strong> splits a \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-suite\">test suite\u003C/span> into parts that run as independent jobs on different machines. In Playwright the option \u003Ccode>--shard=x/y\u003C/code> selects part x of y. With \u003Ccode>fullyParallel: true\u003C/code> the parts are balanced by individual tests; without it, whole files are assigned to shards, and files of different size give shards of different duration. In GitHub Actions the shards are created with a matrix:\u003C/p>\n\u003Cdiv class=\"wiki-code\" data-language=\"yaml\">\u003Cdiv class=\"wiki-code__header\">\u003Cspan class=\"wiki-code__language\">yaml\u003C/span>\u003Cbutton type=\"button\" class=\"wiki-code__copy\" data-wiki-copy data-copied-label=\"Copied\">\u003Csvg class=\"wiki-code__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M216,28H88A12,12,0,0,0,76,40V76H40A12,12,0,0,0,28,88V216a12,12,0,0,0,12,12H168a12,12,0,0,0,12-12V180h36a12,12,0,0,0,12-12V40A12,12,0,0,0,216,28ZM156,204H52V100H156Zm48-48H180V88a12,12,0,0,0-12-12H100V52H204Z\"/>\u003C/svg>\u003Csvg class=\"wiki-code__icon wiki-code__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003Cspan class=\"wiki-code__label\">Copy\u003C/span>\u003Cspan class=\"wiki-code__label wiki-code__label--done\">Copied\u003C/span>\u003C/button>\u003C/div>\u003Cpre>\u003Ccode>jobs:\n  e2e:\n    needs: unit\n    runs-on: ubuntu-latest\n    strategy:\n      fail-fast: false\n      matrix:\n        shardIndex: [1, 2, 3, 4]\n        shardTotal: [4]\n    steps:\n    - uses: actions/checkout@v7\n    - uses: actions/setup-node@v7\n      with:\n        node-version: lts/*\n    - run: npm ci\n    - run: npx playwright install --with-deps\n    - run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>A matrix generates at most 256 jobs per workflow run. The option \u003Ccode>fail-fast\u003C/code> defaults to \u003Ccode>true\u003C/code>: when one job of the matrix fails, the others are cancelled. For test shards it is set to \u003Ccode>false\u003C/code>, so that every shard finishes and reports all failures. In GitLab CI the same is done with the keyword \u003Ccode>parallel\u003C/code> and the variables \u003Ccode>CI_NODE_INDEX\u003C/code> and \u003Ccode>CI_NODE_TOTAL\u003C/code>.\u003C/p>\n\u003Cp>Each shard produces its own report. Playwright merges them with the command \u003Ccode>npx playwright merge-reports\u003C/code> from blob reports; for \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"junit-xml\">JUnit XML\u003C/span>, each shard uploads its own file and the CI system or the receiving system combines them.\u003C/p>\n\u003Ch2 id=\"test-reports\" class=\"wiki-heading\">Test reports\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"test-reports\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>A CI system reads the results of tests from report files. The de facto exchange format is JUnit XML; its structure is described in the article \u003Ca href=\"/en/wiki/test-management/what-is-a-test-management-system\" class=\"wiki-link wiki-link--article\">What is a test management system\u003C/a>. Test frameworks write it with an option or a reporter. In Playwright:\u003C/p>\n\u003Cdiv class=\"wiki-code\" data-language=\"typescript\">\u003Cdiv class=\"wiki-code__header\">\u003Cspan class=\"wiki-code__language\">typescript\u003C/span>\u003Cbutton type=\"button\" class=\"wiki-code__copy\" data-wiki-copy data-copied-label=\"Copied\">\u003Csvg class=\"wiki-code__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M216,28H88A12,12,0,0,0,76,40V76H40A12,12,0,0,0,28,88V216a12,12,0,0,0,12,12H168a12,12,0,0,0,12-12V180h36a12,12,0,0,0,12-12V40A12,12,0,0,0,216,28ZM156,204H52V100H156Zm48-48H180V88a12,12,0,0,0-12-12H100V52H204Z\"/>\u003C/svg>\u003Csvg class=\"wiki-code__icon wiki-code__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003Cspan class=\"wiki-code__label\">Copy\u003C/span>\u003Cspan class=\"wiki-code__label wiki-code__label--done\">Copied\u003C/span>\u003C/button>\u003C/div>\u003Cpre>\u003Ccode>import { defineConfig } from '@playwright/test';\n\nexport default defineConfig({\n  forbidOnly: !!process.env.CI,\n  retries: process.env.CI ? 2 : 0,\n  reporter: process.env.CI\n    ? [['junit', { outputFile: 'results.xml' }], ['blob']]\n    : 'html',\n  use: {\n    trace: 'on-first-retry',\n    screenshot: 'only-on-failure',\n  },\n});\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>The option \u003Ccode>forbidOnly\u003C/code> fails the run if a test marked with \u003Ccode>test.only\u003C/code> was committed by mistake. In pytest the report is produced with \u003Ccode>pytest --junit-xml=results.xml\u003C/code>.\u003C/p>\n\u003Cp>GitLab CI reads JUnit XML from the key \u003Ccode>artifacts:reports:junit\u003C/code>:\u003C/p>\n\u003Cdiv class=\"wiki-code\" data-language=\"yaml\">\u003Cdiv class=\"wiki-code__header\">\u003Cspan class=\"wiki-code__language\">yaml\u003C/span>\u003Cbutton type=\"button\" class=\"wiki-code__copy\" data-wiki-copy data-copied-label=\"Copied\">\u003Csvg class=\"wiki-code__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M216,28H88A12,12,0,0,0,76,40V76H40A12,12,0,0,0,28,88V216a12,12,0,0,0,12,12H168a12,12,0,0,0,12-12V180h36a12,12,0,0,0,12-12V40A12,12,0,0,0,216,28ZM156,204H52V100H156Zm48-48H180V88a12,12,0,0,0-12-12H100V52H204Z\"/>\u003C/svg>\u003Csvg class=\"wiki-code__icon wiki-code__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003Cspan class=\"wiki-code__label\">Copy\u003C/span>\u003Cspan class=\"wiki-code__label wiki-code__label--done\">Copied\u003C/span>\u003C/button>\u003C/div>\u003Cpre>\u003Ccode>e2e:\n  stage: test\n  image: mcr.microsoft.com/playwright:v1.64.1-noble\n  parallel: 4\n  script:\n    - npm ci\n    - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL\n  artifacts:\n    when: always\n    paths:\n      - results.xml\n      - test-results/\n    reports:\n      junit: results.xml\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>\u003Ccode>when: always\u003C/code> uploads the report also when the job fails, which is exactly when it is needed. GitLab shows the results in the "Tests" tab of the pipeline and in the test summary of the merge request. In the merge request it compares the results with the target branch and separates newly failed tests, existing failures and resolved failures. The documentation sets limits: a single JUnit file must be smaller than 30 MB, all files of a job smaller than 100 MB, and of tests with duplicate names only the first is used.\u003C/p>\n\u003Ch2 id=\"retries-of-failed-tests\" class=\"wiki-heading\">Retries of failed tests\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"retries-of-failed-tests\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>A pipeline can rerun failed tests automatically: in Playwright with the option \u003Ccode>retries\u003C/code>, in pytest with the plugin \u003Ccode>pytest-rerunfailures\u003C/code>, in Maven with \u003Ccode>surefire.rerunFailingTestsCount\u003C/code>. A test that passes on a retry is reported as flaky. Retries are usually enabled only on CI, as in the configuration above, with a small number of attempts.\u003C/p>\n\u003Cp>A retry turns an intermittent failure into a pass, including a failure caused by a real defect. For this reason the status "flaky" is kept in the report rather than merged into "passed". Maven Surefire writes failed attempts as \u003Ccode>flakyFailure\u003C/code> elements inside the \u003Ccode>testcase\u003C/code> element; Playwright can fail the run on any \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"flaky-test\">flaky test\u003C/span> with \u003Ccode>--fail-on-flaky-tests\u003C/code>. Causes, detection and \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-quarantine\">quarantine\u003C/span> of flaky tests are described in the article \u003Ca href=\"/en/wiki/test-automation/flaky-tests\" class=\"wiki-link wiki-link--article\">Flaky tests\u003C/a>.\u003C/p>\n\u003Ch2 id=\"artifacts\" class=\"wiki-heading\">Artifacts\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"artifacts\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>A failed test on a CI agent cannot be debugged interactively: the agent is gone when the job ends. The job therefore saves \u003Cstrong>artifacts\u003C/strong>, files that document the run:\u003C/p>\n\u003Cul>\n\u003Cli>screenshots at the moment of failure;\u003C/li>\n\u003Cli>traces with the actions, network requests and DOM snapshots of the test (in Playwright, \u003Ccode>trace: 'on-first-retry'\u003C/code> records a trace on the first retry);\u003C/li>\n\u003Cli>video of the test;\u003C/li>\n\u003Cli>logs of the application and of the \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-environment\">test environment\u003C/span>;\u003C/li>\n\u003Cli>the report files themselves.\u003C/li>\n\u003C/ul>\n\u003Cp>In GitHub Actions artifacts are uploaded with the action \u003Ccode>actions/upload-artifact\u003C/code>:\u003C/p>\n\u003Cdiv class=\"wiki-code\" data-language=\"yaml\">\u003Cdiv class=\"wiki-code__header\">\u003Cspan class=\"wiki-code__language\">yaml\u003C/span>\u003Cbutton type=\"button\" class=\"wiki-code__copy\" data-wiki-copy data-copied-label=\"Copied\">\u003Csvg class=\"wiki-code__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M216,28H88A12,12,0,0,0,76,40V76H40A12,12,0,0,0,28,88V216a12,12,0,0,0,12,12H168a12,12,0,0,0,12-12V180h36a12,12,0,0,0,12-12V40A12,12,0,0,0,216,28ZM156,204H52V100H156Zm48-48H180V88a12,12,0,0,0-12-12H100V52H204Z\"/>\u003C/svg>\u003Csvg class=\"wiki-code__icon wiki-code__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003Cspan class=\"wiki-code__label\">Copy\u003C/span>\u003Cspan class=\"wiki-code__label wiki-code__label--done\">Copied\u003C/span>\u003C/button>\u003C/div>\u003Cpre>\u003Ccode>    - uses: actions/upload-artifact@v7\n      if: ${{ !cancelled() }}\n      with:\n        name: results-${{ matrix.shardIndex }}\n        path: |\n          results.xml\n          test-results/\n        retention-days: 7\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>The condition \u003Ccode>${{ !cancelled() }}\u003C/code> uploads artifacts also after failed steps. The retention period limits storage: artifacts of a failure are needed for days, not for months. GitLab can link a file to a failed test: a line \u003Ccode>[[ATTACHMENT|/path/to/file]]\u003C/code> in \u003Ccode>system-out\u003C/code> of the \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-case\">test case\u003C/span> in JUnit XML appears as a link in the details of the test.\u003C/p>\n\u003Ch2 id=\"results-in-test-runs-of-a-test-management-system\" class=\"wiki-heading\">Results in test runs of a test management system\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"results-in-test-runs-of-a-test-management-system\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>A CI system shows the results of one pipeline. A \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-management-system\">test management system\u003C/span> keeps them as \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-run\">test runs\u003C/span> next to manual testing and over time. The usual mapping:\u003C/p>\n\u003Cul>\n\u003Cli>\u003Cstrong>one test run per pipeline\u003C/strong>: the run is named after the pipeline and records the commit, the branch, the build number and the environment;\u003C/li>\n\u003Cli>\u003Cstrong>one result per test case\u003C/strong>: each automated test is linked to a test case by an identifier that both sides know;\u003C/li>\n\u003Cli>\u003Cstrong>statuses\u003C/strong>: \u003Ccode>failure\u003C/code>, \u003Ccode>error\u003C/code> and \u003Ccode>skipped\u003C/code> of JUnit XML are mapped to the statuses of the run, and flaky results get their own mark;\u003C/li>\n\u003Cli>\u003Cstrong>attachments\u003C/strong>: screenshots and traces are attached to the results of failed tests.\u003C/li>\n\u003C/ul>\n\u003Cp>The identifier of a test case is placed in the test itself. In Playwright this can be an annotation:\u003C/p>\n\u003Cdiv class=\"wiki-code\" data-language=\"typescript\">\u003Cdiv class=\"wiki-code__header\">\u003Cspan class=\"wiki-code__language\">typescript\u003C/span>\u003Cbutton type=\"button\" class=\"wiki-code__copy\" data-wiki-copy data-copied-label=\"Copied\">\u003Csvg class=\"wiki-code__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M216,28H88A12,12,0,0,0,76,40V76H40A12,12,0,0,0,28,88V216a12,12,0,0,0,12,12H168a12,12,0,0,0,12-12V180h36a12,12,0,0,0,12-12V40A12,12,0,0,0,216,28ZM156,204H52V100H156Zm48-48H180V88a12,12,0,0,0-12-12H100V52H204Z\"/>\u003C/svg>\u003Csvg class=\"wiki-code__icon wiki-code__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003Cspan class=\"wiki-code__label\">Copy\u003C/span>\u003Cspan class=\"wiki-code__label wiki-code__label--done\">Copied\u003C/span>\u003C/button>\u003C/div>\u003Cpre>\u003Ccode>import { test, expect } from '@playwright/test';\n\ntest('pay with saved card', {\n  annotation: { type: 'test-case-id', description: 'TC-12' },\n}, async ({ page }) => {\n  await page.goto('https://example.com/checkout');\n  await page.getByRole('button', { name: 'Pay with saved card' }).click();\n  await expect(page.getByRole('heading', { name: 'Payment accepted' })).toBeVisible();\n});\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>The Playwright JUnit reporter writes annotations as properties of the test case:\u003C/p>\n\u003Cdiv class=\"wiki-code\" data-language=\"xml\">\u003Cdiv class=\"wiki-code__header\">\u003Cspan class=\"wiki-code__language\">xml\u003C/span>\u003Cbutton type=\"button\" class=\"wiki-code__copy\" data-wiki-copy data-copied-label=\"Copied\">\u003Csvg class=\"wiki-code__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M216,28H88A12,12,0,0,0,76,40V76H40A12,12,0,0,0,28,88V216a12,12,0,0,0,12,12H168a12,12,0,0,0,12-12V180h36a12,12,0,0,0,12-12V40A12,12,0,0,0,216,28ZM156,204H52V100H156Zm48-48H180V88a12,12,0,0,0-12-12H100V52H204Z\"/>\u003C/svg>\u003Csvg class=\"wiki-code__icon wiki-code__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003Cspan class=\"wiki-code__label\">Copy\u003C/span>\u003Cspan class=\"wiki-code__label wiki-code__label--done\">Copied\u003C/span>\u003C/button>\u003C/div>\u003Cpre>\u003Ccode><testcase name="pay with saved card" classname="checkout.spec.ts" time="1.62">\n  <properties>\n    <property name="test-case-id" value="TC-12"/>\n  </properties>\n</testcase>\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>In pytest the same is done with the fixture \u003Ccode>record_property\u003C/code>. The results reach the test management system either as an imported report file or through its API, called by a job at the end of the pipeline. The structure of a test run and the statuses of results are described in the article \u003Ca href=\"/en/wiki/test-management/test-run\" class=\"wiki-link wiki-link--article\">Test run\u003C/a>.\u003C/p>\n\u003Ch2 id=\"test-selection-and-impact-analysis\" class=\"wiki-heading\">Test selection and impact analysis\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"test-selection-and-impact-analysis\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>When the full suite becomes too long for every change, the pipeline runs only the tests affected by the change. \u003Cstrong>Test impact analysis\u003C/strong> determines which subset of tests to run for a given set of changes. Paul Hammant describes on Fowler's site (2017) the usual mechanism: a map from tests to the source files they execute, built from code coverage while each test runs separately, stored in version control and updated by CI.\u003C/p>\n\u003Cp>The map has gaps. Changes of configuration, resource files and data are not visible in code coverage, and new tests are not in the map until it is rebuilt. For this reason selection is combined with a full run on a schedule. A simpler form of selection is built into some runners: Playwright's \u003Ccode>--only-changed\u003C/code> runs only the test files changed relative to a Git reference.\u003C/p>\n\u003Ch2 id=\"environments-and-test-data-in-ci\" class=\"wiki-heading\">Environments and test data in CI\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"environments-and-test-data-in-ci\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>Tests in CI run in an environment that the pipeline prepares or that exists permanently. Fowler lists testing in a clone of the production environment among the practices of CI. Common approaches:\u003C/p>\n\u003Cul>\n\u003Cli>dependencies (database, message queue, mocks of external services) are started in containers for the duration of the job, so every pipeline gets a clean state;\u003C/li>\n\u003Cli>a deployed test environment is shared between pipelines, and runs that use it are limited by a concurrency group; in GitHub Actions \u003Ccode>concurrency\u003C/code> with \u003Ccode>cancel-in-progress: true\u003C/code> cancels an older run in the same group;\u003C/li>\n\u003Cli>test data is created by the tests or loaded by the pipeline before the tests, and each shard uses its own data, so that parallel jobs do not interfere;\u003C/li>\n\u003Cli>secrets, such as tokens of test accounts, come from the secret store of the CI system rather than from the repository.\u003C/li>\n\u003C/ul>\n\u003Cp>The environment is recorded with the results: the version of the build, the address of the environment, the browser and its version. Without it a failed result cannot be reproduced.\u003C/p>\n\u003Ch2 id=\"ci-continuous-delivery-and-continuous-deployment\" class=\"wiki-heading\">CI, continuous delivery and continuous deployment\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"ci-continuous-delivery-and-continuous-deployment\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The table compares three related practices.\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-4\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>\u003C/th>\n\u003Cth>Continuous integration\u003C/th>\n\u003Cth>Continuous delivery\u003C/th>\n\u003Cth>Continuous deployment\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Goal\u003C/td>\n\u003Ctd>Changes are integrated and checked often\u003C/td>\n\u003Ctd>The product can be released at any time\u003C/td>\n\u003Ctd>Every change is released automatically\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>End of the pipeline\u003C/td>\n\u003Ctd>Tested build\u003C/td>\n\u003Ctd>Release candidate ready for production\u003C/td>\n\u003Ctd>Change in production\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Release decision\u003C/td>\n\u003Ctd>Not part of CI\u003C/td>\n\u003Ctd>Made by people\u003C/td>\n\u003Ctd>Made by the pipeline\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Role of automated tests\u003C/td>\n\u003Ctd>Check every change\u003C/td>\n\u003Ctd>Gate the release candidate\u003C/td>\n\u003Ctd>The only gate before production\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>Continuous delivery was described in the book "Continuous Delivery" by Jez Humble and David Farley (2010). Fowler defines it as building software so that it can be released to production at any time. Continuous deployment goes one step further: every change that passes the pipeline is put into production automatically. Continuous deployment requires continuous delivery, and both require CI.\u003C/p>\n\u003Cp>The combined term CI/CD covers the whole path from a commit to a release. For automated tests the difference is in the consequence of a pass: in CI it allows a merge, in continuous delivery it allows a release, in continuous deployment it performs it.\u003C/p>\n\u003Ch2 id=\"terms\" class=\"wiki-heading\">Terms\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"terms\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cdiv class=\"wiki-table wiki-table--cols-2\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Term\u003C/th>\n\u003Cth>Meaning\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Continuous integration\u003C/td>\n\u003Ctd>Practice in which changes are merged into a shared mainline at least daily and every integration is verified by an automated build with tests\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Pipeline\u003C/td>\n\u003Ctd>Sequence of jobs that a CI system executes for one event: build, tests, publication, deployment\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Quality gate\u003C/td>\n\u003Ctd>Stage of a pipeline whose failure stops the pipeline\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Sharding\u003C/td>\n\u003Ctd>Splitting a test suite into parts that run as independent jobs on different machines\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Artifact\u003C/td>\n\u003Ctd>File saved by a CI job: report, screenshot, trace, log\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Test impact analysis\u003C/td>\n\u003Ctd>Technique that selects the subset of tests to run for a given set of changes\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Continuous delivery\u003C/td>\n\u003Ctd>Practice in which the product is kept in a state that can be released to production at any time\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Continuous deployment\u003C/td>\n\u003Ctd>Practice in which every change that passes the pipeline is released to production automatically\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Ch2 id=\"sources\" class=\"wiki-heading\">Sources\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"sources\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cul>\n\u003Cli>\u003Ca href=\"https://martinfowler.com/articles/continuousIntegration.html\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Continuous Integration, Martin Fowler, revised 18 January 2024\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://martinfowler.com/bliki/ContinuousDelivery.html\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Continuous Delivery, Martin Fowler, 30 May 2013\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://martinfowler.com/articles/rise-test-impact-analysis.html\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">The Rise of Test Impact Analysis, Paul Hammant, 2017\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://www.istqb.org/wp-content/uploads/2024/11/ISTQB_CTAL-TAE_Syllabus_v2.0.pdf\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">ISTQB Certified Tester Advanced Level Test Automation Engineering Syllabus v2.0, ISTQB, 2024\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Events that trigger workflows, GitHub Docs\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Workflow syntax for GitHub Actions, GitHub Docs\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://docs.gitlab.com/ci/testing/unit_test_reports/\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Unit test reports, GitLab Docs\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://playwright.dev/docs/test-sharding\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Sharding, Playwright documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://playwright.dev/docs/test-reporters\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Reporters, Playwright documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://playwright.dev/docs/ci\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Continuous Integration, Playwright documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://docs.pytest.org/en/stable/how-to/output.html\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Managing pytest's output, pytest documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Rerun failing tests, Maven Surefire Plugin\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003C/ul>\n",[10,14,17,20,23,26,29,32,35,38,41,44,47],{"id":11,"level":12,"text":13},"the-problem-ci-solves",2,"The problem CI solves",{"id":15,"level":12,"text":16},"pipeline-triggers","Pipeline triggers",{"id":18,"level":12,"text":19},"stages-by-speed","Stages by speed",{"id":21,"level":12,"text":22},"parallelism-and-sharding","Parallelism and sharding",{"id":24,"level":12,"text":25},"test-reports","Test reports",{"id":27,"level":12,"text":28},"retries-of-failed-tests","Retries of failed tests",{"id":30,"level":12,"text":31},"artifacts","Artifacts",{"id":33,"level":12,"text":34},"results-in-test-runs-of-a-test-management-system","Results in test runs of a test management system",{"id":36,"level":12,"text":37},"test-selection-and-impact-analysis","Test selection and impact analysis",{"id":39,"level":12,"text":40},"environments-and-test-data-in-ci","Environments and test data in CI",{"id":42,"level":12,"text":43},"ci-continuous-delivery-and-continuous-deployment","CI, continuous delivery and continuous deployment",{"id":45,"level":12,"text":46},"terms","Terms",{"id":48,"level":12,"text":49},"sources","Sources",2371,1793718940562]