Test automation levels and the test pyramid

Test automation levels are groups of automated tests that differ in how large a part of the system a single test exercises: a unit test checks one component in isolation, an integration test checks the interaction of components or systems, and an end-to-end test checks the whole deployed system through its external interfaces. The test pyramid is a model of how automated tests are distributed between these levels: many small, fast tests at the bottom and few large, slow tests at the top.

The pyramid comes from Mike Cohn. According to Martin Fowler, Cohn drew it in conversations with Lisa Crispin in 2003-2004. Cohn published it as the "test automation pyramid" in the book Succeeding with Agile (Addison-Wesley, 2009). Fowler's bliki entry TestPyramid (2012) and Ham Vocke's article "The Practical Test Pyramid" (2018) spread the model further. The ISTQB Foundation Level syllabus (version 4.0.1) describes the test pyramid in section 5.1.6 and the test levels in section 2.2.1.

The problem the test pyramid solves

The same behavior can be checked by tests of different size. A rule that rejects an expired payment card can be checked by calling one function, by sending a request to the payment service, or by paying with an expired card in a browser. All three tests can find the same defect. They differ in run time, in the cost of keeping them working and in how precisely a failure points to the cause.

A test suite built mostly from tests that drive the user interface has known problems. Fowler lists them: such tests are brittle, expensive to write and slow to run; a change to the interface can break many of them at once; and they are more prone to non-deterministic results.

The test pyramid answers two questions: which kinds of tests to write, and in what proportion. Vocke reduces it to two rules of thumb: write tests with different granularity, and the higher the level, the fewer tests there should be.

History

The table below lists the milestones of the model.

Date	Event
2003-2004	Mike Cohn draws the pyramid in conversations with Lisa Crispin (according to Fowler)
2006	Jason Huggins comes to the same idea independently (according to Fowler)
20 October 2009	Succeeding with Agile by Mike Cohn is published, with the test automation pyramid of unit, service and UI tests
January 2012	Alister Scott describes the "software testing ice-cream cone" anti-pattern
1 May 2012	Martin Fowler publishes the bliki entry TestPyramid
6 February 2018	Kent C. Dodds shares the testing trophy
26 February 2018	Ham Vocke publishes "The Practical Test Pyramid" on martinfowler.com
9 May 2023	ISTQB releases CTFL v4.0
2024	CTFL v4.0.1, with the test pyramid in section 5.1.6

Test levels and automation levels

The ISTQB syllabus defines five test levels. Test levels are groups of test activities that are organized and managed together. The table shows the five levels and the automation level that usually corresponds to each.

ISTQB test level	Focus	Usual automation level
Component testing (unit testing)	Components in isolation	Unit tests
Component integration testing	Interfaces and interactions between components	Integration tests
System testing	Behavior and capabilities of the whole system	End-to-end tests
System integration testing	Interfaces with other systems and external services	Integration or end-to-end tests
Acceptance testing	Validation and readiness for deployment	Any level

A test level describes an activity of the test process: who performs it, against which basis, with which objectives. An automation level describes a technical property of one automated test: what it starts and what it replaces with test doubles. Component testing maps directly to unit tests. An acceptance test can be automated at any level: Vocke notes that a check of a feature from the user's point of view does not have to run through the user interface.

Unit tests

A unit test calls a function, a class or a module directly, in the process of the test framework. Slow or uncontrolled dependencies, such as a database, the network or the clock, are replaced with test doubles. The syllabus notes that component testing often requires test harnesses or unit test frameworks and is normally performed by developers in their development environments.

A unit test of the rule "an expired card is rejected" passes a card with a past expiry date to the validation function and checks the returned error. It needs no server, no database and no browser. Its failure points to one function.

Integration tests

The term "integration test" has at least two meanings, as Fowler notes. A narrow integration test exercises only the part of a service that talks to another component: a database, a message broker, another service. Its scope stays close to that of a unit test. A broad integration test needs live versions of all services and a test environment with network access.

The ISTQB levels follow a similar split. Component integration testing depends heavily on the integration strategy: bottom-up, top-down or big-bang. System integration testing requires a test environment that is preferably similar to the operational environment.

An integration test of the payment rule sends a request with an expired card to the payment service running with a real database, and checks the HTTP status and the error code in the response. A failure points to the service, its database schema or the mapping between them.

End-to-end tests

An end-to-end test runs against the deployed system and goes through an external interface: the user interface or the public API. In the payment example, the test opens the shop in a browser, logs in, adds goods to the cart, pays with an expired card and checks the message on the screen.

Such a test is the closest to how the product is used. It also depends on the most things at once: the build, the configuration, the test environment, the test data, the network and the browser. How end-to-end tests find elements on a page is described in the article Locators in UI tests.

Cost and speed of each level

The syllabus states the general rule: the higher the layer, the lower the test granularity, test isolation and test execution time. Tests in the bottom layer are small, isolated and fast. Tests in the top layer are generally slower. The table compares the levels by the factors that make up the cost of a test.

Factor	Unit test	Integration test	End-to-end test
What runs	Code in the test process	A service with its real dependencies	The whole deployed system
Slowest operation	Function calls in memory	Database queries, network calls	Page loads, rendering, all services
Environment needed	None	Database, broker, service containers	Full test environment
Failure localization	One function or class	One service or interface	Anywhere in the system
Reasons to fail without a defect	Few	Environment, test data	Environment, data, timing, interface changes
What it cannot find	Defects in interactions	Defects in the complete user flow	Little; it is the closest to real use

Exact numbers depend on the product and the infrastructure; there are no universal figures. The difference comes from the operations themselves. A function call in memory takes far less time than a database query over the network, and a query takes less than loading a page that triggers many queries.

Some build tools encode this difference explicitly. In Bazel a test target has a size attribute with the values small, medium, large and enormous. The size sets the default timeout: 1 minute, 5 minutes, 15 minutes and 60 minutes respectively, and the resources reserved for local runs.

Execution time is only part of the cost. A failure of an end-to-end test requires an investigation: the product, the environment, the data or the test itself can be at fault. Tests that fail without a defect in the product are described in the article Flaky tests.

The test pyramid

The syllabus describes the pyramid as a model showing that tests may have different granularity. The layers represent groups of tests. Tests in the bottom layer check a small piece of functionality each, so many of them are needed to achieve reasonable coverage. Tests in the top layer check a large piece of functionality each, so a few of them are enough.

            /\
           /  \        End-to-end tests: few, slow, broad
          /----\
         /      \      Integration tests
        /--------\
       /          \    Unit tests: many, fast, isolated
      +------------+

The syllabus notes that the number and names of the layers differ. Cohn's original pyramid has three layers: unit tests, service tests and UI tests. Another common variant has unit (component) tests, integration (component integration) tests and end-to-end tests.

The middle layer is the least obvious. Cohn defines a service as something the application does in response to some input or set of inputs, and service tests check these services without the user interface. Fowler calls such tests subcutaneous: they act through the service layer or the API of the application just under the user interface. They give many of the benefits of end-to-end tests without the complexity of the user interface.

Vocke warns against taking the names of the layers too literally: the pyramid is simplistic and can be misleading. The shape does not prescribe exact proportions.

The ice cream cone anti-pattern

The ice cream cone is the inverted test pyramid: most checks are manual tests and tests through the user interface, and few are unit tests. Alister Scott described it in his blog WatirMelon in January 2012. He wrote that organizations often create inverted testing pyramids. His own pyramid has a cloud of manual session-based tests on top, and when it is inverted, the shape becomes an ice cream cone.

Fowler's TestPyramid entry describes how such a suite appears. A team automates tests with record and playback tools: a tester records actions in the user interface, and the tool replays them and compares the results. Fowler notes that this works well at first. The suite then grows at the top level and breaks with changes to the interface. Fowler calls the result an ice cream cone.

The anti-pattern describes the proportion, not the existence of end-to-end tests. A suite with a few end-to-end tests for the main user flows on top of many lower-level tests is still a pyramid.

Other shapes

The testing trophy was shared by Kent C. Dodds on 6 February 2018 and described in his article of 3 June 2021. It has four layers, from bottom to top: static checks (type checkers and linters), unit tests, integration tests and end-to-end tests. The widest layer is integration. Dodds takes the idea from Guillermo Rauch's advice of 10 December 2016: "Write tests. Not too many. Mostly integration."

The table compares the two shapes.

	Test pyramid	Testing trophy
Author, year	Mike Cohn, 2009	Kent C. Dodds, 2018
Layers	Unit, service, UI (original)	Static, unit, integration, end-to-end
Widest layer	Unit tests	Integration tests
Argument	Lower tests are faster and cheaper	Return on investment: confidence for time spent
Stated scope	General	One codebase; Dodds applied it to monoliths

Both shapes put few end-to-end tests at the top, and both relate the number of tests to their cost. They differ in where most of the effort goes. Dodds explains his support for Rauch's advice by his experience as a user interface engineer, and states that the classification itself matters less than the goal.

In both models the meaning of "integration" has to be agreed within the team. Dodds defines integration tests as tests of several units working together, which is closer to Fowler's narrow integration tests than to broad ones.

Choosing the level for a check

The usual rule follows from the pyramid: a check belongs to the lowest level at which it can find the defect it is aimed at. Vocke formulates two consequences. If a higher-level test finds an error and no lower-level test fails, a lower-level test is missing and should be written. And a high-level test that only repeats checks already made at lower levels adds cost without adding confidence.

A check needs a higher level when its defect exists only there. Wiring, configuration, the database schema and the behavior of a real browser cannot be checked by a unit test, because it replaces exactly these parts with test doubles.

The table distributes the checks of one feature, a promo code at checkout, between the levels.

Check	Level	Why this level
Promo code format is validated	Unit	Pure function of the input
Discount at the boundaries of the order amount	Unit	Many input values, each test is cheap
Expired promo code is rejected	Unit	The clock is replaced with a test double
Order with a discount is saved and read back	Integration	Depends on the real database schema
Request for a discount follows the agreed API	Contract test	Two services, each tested separately
User applies a code and sees the new total	End-to-end	Whole flow through the browser, one test

The values for the second row are chosen by test design, described in the article Equivalence partitioning and boundary values. The pyramid only says that such numerous checks should be cheap.

Contract tests between services

The interaction of two services can be checked without deploying them together. A contract test checks each side of an interface separately against a shared description of the requests and responses. In consumer-driven contracts, a pattern described by Ian Robinson in 2006, consumers state their expectations of a provider, and the provider's obligations are the sum of these expectations.

Pact implements this pattern. The consumer's tests send real requests to a mock provider created by Pact. The mock checks each request and returns the expected response. When the consumer tests pass, Pact writes a pact file that describes each interaction. In provider verification, Pact replays every recorded request against the real provider and compares its response with the minimal expected response.

Consumer tests                        Provider verification
 |                                     |
 +-- request --> Pact mock provider    +-- recorded request --> real provider
 |               |                     |                         |
 |               +-- expected response |                         +-- actual response
 |                                     |                               |
 +--> pact file -----------------------+--> compared with expected <---+

Only the parts of the provider that consumers use are checked, so the provider can change unused behavior freely. Contract tests take over part of the work of broad integration tests without a full test environment. They do not check the deployed system, so a few end-to-end tests remain.

Levels, coverage and continuous integration

Code coverage is measured mostly for unit and integration tests, whose process can be instrumented. Requirements coverage is computed from links between requirements and tests and is typical for system and acceptance tests. The two measures are compared in the article Requirements traceability and coverage.

The level of a test also determines where it can run in a continuous integration (CI) pipeline. Unit tests need no environment and can run on every change. Integration tests need their dependencies, for example started in containers for the duration of the job. End-to-end tests need a deployed system. How tests are triggered by pipeline events, run in parallel and reported is described in the article Automated tests in CI.

Terms

Term	Meaning
Test automation level	Group of automated tests that exercise a part of the system of similar size
Unit test	Automated test that checks one component in isolation, in the test process
Integration test	Automated test that checks the interaction of components or of a system with other systems
End-to-end test	Automated test that checks the whole deployed system through its user interface or public API
Test pyramid	Model of many small, fast tests at the bottom and few large, slow tests at the top
Ice cream cone	Anti-pattern of an inverted pyramid: mostly manual and UI tests, few unit tests
Testing trophy	Model of Kent C. Dodds with static, unit, integration and end-to-end layers, integration the widest
Contract test	Test that checks each side of an interface separately against a shared description of requests and responses
Consumer-driven contract	Contract of a provider formed from the expectations of its consumers