Automated tests in CI

Continuous integration (CI) is a development practice in which the members of a team merge their changes into a shared mainline at least daily, and every integration is verified by an automated build that includes tests. Automated tests are the part of this build that decides whether a change can be accepted. A CI system starts them on events in the code repository and on a schedule, collects their results and reports them back to the change, the team and other systems.

Continuous integration was developed by Kent Beck as a practice of Extreme Programming in the 1990s. Martin Fowler published the article "Continuous Integration" in 2000 and revised it in January 2024; it lists the practices, among them a self-testing build, a build on every push to the mainline and a fast build. The term itself is sometimes credited to Grady Booch, who, according to Fowler, mentioned it only in passing in a book on object-oriented design. The ISTQB Advanced Level Test Automation Engineering syllabus (version 2.0, 2024) devotes a chapter to the integration of automated tests into CI/CD pipelines: which test levels run at which stage and which of them act as quality gates.

The problem CI solves

Without CI, changes are integrated rarely and in large portions. Conflicts and defects accumulate, and their cause is hard to find because many changes have been made since the last working state. Tests that are run by hand before a release are run late, on an unknown combination of changes, and sometimes not at all.

CI shortens the distance between a change and its check. Every push or merge request starts the build and the tests, and the author gets the result while the change is still fresh. Fowler formulates the requirement for speed through the Extreme Programming guideline of a ten-minute build. A failed build is fixed immediately, so that the mainline stays in a working state.

The levels of automated tests (unit, integration, end-to-end) and their cost are described in the article Test automation levels and the test pyramid. This article describes how these tests are run in CI and how their results are used.

Pipeline triggers

A pipeline is a sequence of jobs that a CI system executes for one event: building, running tests, publishing artifacts, deploying. A pipeline is started by a trigger. The usual triggers and the tests they run:

In GitHub Actions the triggers are declared in the key on of a workflow:

name: Tests

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]
  schedule:
    - cron: '0 2 * * *'
  workflow_dispatch:
    inputs:
      suite:
        description: 'Test suite to run'
        required: true
        default: 'smoke'
        type: choice
        options:
        - smoke
        - regression

jobs:
  unit:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v7
    - uses: actions/setup-node@v7
      with:
        node-version: lts/*
    - run: npm ci
    - run: npm test

The GitHub Actions documentation defines the details of each trigger. By default, pull_request fires for the activity types opened, synchronize and reopened, and the workflow runs on the merge commit of the request (refs/pull/<number>/merge), so the tests check the result of the merge rather than the branch alone. A schedule uses POSIX cron syntax, runs in UTC by default on the latest commit of the default branch, cannot run more often than every five minutes and can be delayed when the load is high. In public repositories scheduled workflows are disabled after 60 days without activity. workflow_dispatch works only if the workflow file is on the default branch; its inputs can be of the types boolean, choice, number, environment and string.

Stages by speed

Tests in a pipeline are ordered from fast to slow. A fast stage fails early and does not spend resources on slow stages for a change that is already broken. Fowler describes this as a deployment pipeline: the commit build runs on every push and must be fast, and secondary builds run slower tests, for example against a real database, without blocking every commit.

The CTAL-TAE syllabus distributes test levels between stages:

• component (unit) tests run in the build phase and act as quality gates of the pipeline; component integration tests run alongside them;
• system integration tests are often quality gates of a continuous delivery pipeline;
• system tests can run in a continuous deployment pipeline as the last quality gate before the product is delivered;
• system and acceptance tests run either in the deployment phase, where a failure can roll back the deployment, or in a separate pipeline started by a successful deployment.

A typical order, with illustrative durations:

push / pull request
 |
 +-- build, static checks               1-2 min
 +-- unit tests                         2-5 min    quality gate
 +-- integration tests                  5-10 min   quality gate
 +-- deploy to test environment
 +-- end-to-end tests, 4 shards         10-15 min  quality gate
 +-- report: JUnit XML, artifacts
nightly schedule
 +-- full regression, performance tests

In GitHub Actions the order of jobs is set with needs: a job starts only after the jobs it needs have completed successfully, and if one of them fails or is skipped, the dependent jobs are skipped.

Parallelism and sharding

Parallelism inside one machine and distribution between machines are two different levels. Playwright runs test files in parallel worker processes; the number of workers defaults to half of the logical CPU cores. Its CI guide recommends one worker on CI agents for stability and reproducibility, and sharding for wider parallelization.

Sharding splits a test suite into parts that run as independent jobs on different machines. In Playwright the option --shard=x/y selects part x of y. With fullyParallel: true the parts are balanced by individual tests; without it, whole files are assigned to shards, and files of different size give shards of different duration. In GitHub Actions the shards are created with a matrix:

jobs:
  e2e:
    needs: unit
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]
    steps:
    - uses: actions/checkout@v7
    - uses: actions/setup-node@v7
      with:
        node-version: lts/*
    - run: npm ci
    - run: npx playwright install --with-deps
    - run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}

A matrix generates at most 256 jobs per workflow run. The option fail-fast defaults to true: when one job of the matrix fails, the others are cancelled. For test shards it is set to false, so that every shard finishes and reports all failures. In GitLab CI the same is done with the keyword parallel and the variables CI_NODE_INDEX and CI_NODE_TOTAL.

Each shard produces its own report. Playwright merges them with the command npx playwright merge-reports from blob reports; for JUnit XML, each shard uploads its own file and the CI system or the receiving system combines them.

Test reports

A CI system reads the results of tests from report files. The de facto exchange format is JUnit XML; its structure is described in the article What is a test management system. Test frameworks write it with an option or a reporter. In Playwright:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  reporter: process.env.CI
    ? [['junit', { outputFile: 'results.xml' }], ['blob']]
    : 'html',
  use: {
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
  },
});

The option forbidOnly fails the run if a test marked with test.only was committed by mistake. In pytest the report is produced with pytest --junit-xml=results.xml.

GitLab CI reads JUnit XML from the key artifacts:reports:junit:

e2e:
  stage: test
  image: mcr.microsoft.com/playwright:v1.64.1-noble
  parallel: 4
  script:
    - npm ci
    - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  artifacts:
    when: always
    paths:
      - results.xml
      - test-results/
    reports:
      junit: results.xml

when: always uploads the report also when the job fails, which is exactly when it is needed. GitLab shows the results in the "Tests" tab of the pipeline and in the test summary of the merge request. In the merge request it compares the results with the target branch and separates newly failed tests, existing failures and resolved failures. The documentation sets limits: a single JUnit file must be smaller than 30 MB, all files of a job smaller than 100 MB, and of tests with duplicate names only the first is used.

Retries of failed tests

A pipeline can rerun failed tests automatically: in Playwright with the option retries, in pytest with the plugin pytest-rerunfailures, in Maven with surefire.rerunFailingTestsCount. A test that passes on a retry is reported as flaky. Retries are usually enabled only on CI, as in the configuration above, with a small number of attempts.

A retry turns an intermittent failure into a pass, including a failure caused by a real defect. For this reason the status "flaky" is kept in the report rather than merged into "passed". Maven Surefire writes failed attempts as flakyFailure elements inside the testcase element; Playwright can fail the run on any flaky test with --fail-on-flaky-tests. Causes, detection and quarantine of flaky tests are described in the article Flaky tests.

Artifacts

A failed test on a CI agent cannot be debugged interactively: the agent is gone when the job ends. The job therefore saves artifacts, files that document the run:

• screenshots at the moment of failure;
• traces with the actions, network requests and DOM snapshots of the test (in Playwright, trace: 'on-first-retry' records a trace on the first retry);
• video of the test;
• logs of the application and of the test environment;
• the report files themselves.

In GitHub Actions artifacts are uploaded with the action actions/upload-artifact:

    - uses: actions/upload-artifact@v7
      if: ${{ !cancelled() }}
      with:
        name: results-${{ matrix.shardIndex }}
        path: |
          results.xml
          test-results/
        retention-days: 7

The condition ${{ !cancelled() }} uploads artifacts also after failed steps. The retention period limits storage: artifacts of a failure are needed for days, not for months. GitLab can link a file to a failed test: a line [[ATTACHMENT|/path/to/file]] in system-out of the test case in JUnit XML appears as a link in the details of the test.

Results in test runs of a test management system

A CI system shows the results of one pipeline. A test management system keeps them as test runs next to manual testing and over time. The usual mapping:

• one test run per pipeline: the run is named after the pipeline and records the commit, the branch, the build number and the environment;
• one result per test case: each automated test is linked to a test case by an identifier that both sides know;
• statuses: failure, error and skipped of JUnit XML are mapped to the statuses of the run, and flaky results get their own mark;
• attachments: screenshots and traces are attached to the results of failed tests.

The identifier of a test case is placed in the test itself. In Playwright this can be an annotation:

import { test, expect } from '@playwright/test';

test('pay with saved card', {
  annotation: { type: 'test-case-id', description: 'TC-12' },
}, async ({ page }) => {
  await page.goto('https://example.com/checkout');
  await page.getByRole('button', { name: 'Pay with saved card' }).click();
  await expect(page.getByRole('heading', { name: 'Payment accepted' })).toBeVisible();
});

The Playwright JUnit reporter writes annotations as properties of the test case:

<testcase name="pay with saved card" classname="checkout.spec.ts" time="1.62">
  <properties>
    <property name="test-case-id" value="TC-12"/>
  </properties>
</testcase>

In pytest the same is done with the fixture record_property. The results reach the test management system either as an imported report file or through its API, called by a job at the end of the pipeline. The structure of a test run and the statuses of results are described in the article Test run.

Test selection and impact analysis

When the full suite becomes too long for every change, the pipeline runs only the tests affected by the change. Test impact analysis determines which subset of tests to run for a given set of changes. Paul Hammant describes on Fowler's site (2017) the usual mechanism: a map from tests to the source files they execute, built from code coverage while each test runs separately, stored in version control and updated by CI.

The map has gaps. Changes of configuration, resource files and data are not visible in code coverage, and new tests are not in the map until it is rebuilt. For this reason selection is combined with a full run on a schedule. A simpler form of selection is built into some runners: Playwright's --only-changed runs only the test files changed relative to a Git reference.

Environments and test data in CI

Tests in CI run in an environment that the pipeline prepares or that exists permanently. Fowler lists testing in a clone of the production environment among the practices of CI. Common approaches:

• dependencies (database, message queue, mocks of external services) are started in containers for the duration of the job, so every pipeline gets a clean state;
• a deployed test environment is shared between pipelines, and runs that use it are limited by a concurrency group; in GitHub Actions concurrency with cancel-in-progress: true cancels an older run in the same group;
• test data is created by the tests or loaded by the pipeline before the tests, and each shard uses its own data, so that parallel jobs do not interfere;
• secrets, such as tokens of test accounts, come from the secret store of the CI system rather than from the repository.

The environment is recorded with the results: the version of the build, the address of the environment, the browser and its version. Without it a failed result cannot be reproduced.

CI, continuous delivery and continuous deployment

The table compares three related practices.

Continuous delivery was described in the book "Continuous Delivery" by Jez Humble and David Farley (2010). Fowler defines it as building software so that it can be released to production at any time. Continuous deployment goes one step further: every change that passes the pipeline is put into production automatically. Continuous deployment requires continuous delivery, and both require CI.

The combined term CI/CD covers the whole path from a commit to a release. For automated tests the difference is in the consequence of a pass: in CI it allows a merge, in continuous delivery it allows a release, in continuous deployment it performs it.

Terms

Sources

• Continuous Integration, Martin Fowler, revised 18 January 2024(opens in a new tab)
• Continuous Delivery, Martin Fowler, 30 May 2013(opens in a new tab)
• The Rise of Test Impact Analysis, Paul Hammant, 2017(opens in a new tab)
• ISTQB Certified Tester Advanced Level Test Automation Engineering Syllabus v2.0, ISTQB, 2024(opens in a new tab)
• Events that trigger workflows, GitHub Docs(opens in a new tab)
• Workflow syntax for GitHub Actions, GitHub Docs(opens in a new tab)
• Unit test reports, GitLab Docs(opens in a new tab)
• Sharding, Playwright documentation(opens in a new tab)
• Reporters, Playwright documentation(opens in a new tab)
• Continuous Integration, Playwright documentation(opens in a new tab)
• Managing pytest's output, pytest documentation(opens in a new tab)
• Rerun failing tests, Maven Surefire Plugin(opens in a new tab)