[{"data":1,"prerenderedAt":58},["ShallowReactive",2],{"wiki-test-automation-automation-levels-and-test-pyramid-en":3},{"meta":4,"html":8,"headings":9,"words":57},{"title":5,"description":6,"published":7,"updated":7},"Test automation levels and the test pyramid","Unit, integration and end-to-end tests: the cost and speed of each level, the test pyramid and the ice cream cone anti-pattern.","2026-11-03","\u003Cp>\u003Cstrong>Test automation levels\u003C/strong> are groups of automated tests that differ in how large a part of the system a single test exercises: a \u003Cstrong>unit test\u003C/strong> checks one component in isolation, an \u003Cstrong>integration test\u003C/strong> checks the interaction of components or systems, and an \u003Cstrong>end-to-end test\u003C/strong> checks the whole deployed system through its external interfaces. The \u003Cstrong>test pyramid\u003C/strong> is a model of how automated tests are distributed between these levels: many small, fast tests at the bottom and few large, slow tests at the top.\u003C/p>\n\u003Cp>The pyramid comes from Mike Cohn. According to Martin Fowler, Cohn drew it in conversations with Lisa Crispin in 2003-2004. Cohn published it as the "test automation pyramid" in the book Succeeding with Agile (Addison-Wesley, 2009). Fowler's bliki entry TestPyramid (2012) and Ham Vocke's article "The Practical Test Pyramid" (2018) spread the model further. The ISTQB Foundation Level syllabus (version 4.0.1) describes the test pyramid in section 5.1.6 and the test levels in section 2.2.1.\u003C/p>\n\u003Ch2 id=\"the-problem-the-test-pyramid-solves\" class=\"wiki-heading\">The problem the test pyramid solves\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"the-problem-the-test-pyramid-solves\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The same behavior can be checked by tests of different size. A rule that rejects an expired payment card can be checked by calling one function, by sending a request to the payment service, or by paying with an expired card in a browser. All three tests can find the same \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"defect\">defect\u003C/span>. They differ in run time, in the cost of keeping them working and in how precisely a failure points to the cause.\u003C/p>\n\u003Cp>A \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-suite\">test suite\u003C/span> built mostly from tests that drive the user interface has known problems. Fowler lists them: such tests are brittle, expensive to write and slow to run; a change to the interface can break many of them at once; and they are more prone to non-deterministic results.\u003C/p>\n\u003Cp>The test pyramid answers two questions: which kinds of tests to write, and in what proportion. Vocke reduces it to two rules of thumb: write tests with different granularity, and the higher the level, the fewer tests there should be.\u003C/p>\n\u003Ch2 id=\"history\" class=\"wiki-heading\">History\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"history\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The table below lists the milestones of the model.\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-2\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Date\u003C/th>\n\u003Cth>Event\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>2003-2004\u003C/td>\n\u003Ctd>Mike Cohn draws the pyramid in conversations with Lisa Crispin (according to Fowler)\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>2006\u003C/td>\n\u003Ctd>Jason Huggins comes to the same idea independently (according to Fowler)\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>20 October 2009\u003C/td>\n\u003Ctd>Succeeding with Agile by Mike Cohn is published, with the test automation pyramid of unit, service and UI tests\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>January 2012\u003C/td>\n\u003Ctd>Alister Scott describes the "software testing ice-cream cone" anti-pattern\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>1 May 2012\u003C/td>\n\u003Ctd>Martin Fowler publishes the bliki entry TestPyramid\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>6 February 2018\u003C/td>\n\u003Ctd>Kent C. Dodds shares the testing trophy\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>26 February 2018\u003C/td>\n\u003Ctd>Ham Vocke publishes "The Practical Test Pyramid" on martinfowler.com\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>9 May 2023\u003C/td>\n\u003Ctd>ISTQB releases CTFL v4.0\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>2024\u003C/td>\n\u003Ctd>CTFL v4.0.1, with the test pyramid in section 5.1.6\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Ch2 id=\"test-levels-and-automation-levels\" class=\"wiki-heading\">Test levels and automation levels\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"test-levels-and-automation-levels\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The ISTQB syllabus defines five test levels. Test levels are groups of test activities that are organized and managed together. The table shows the five levels and the automation level that usually corresponds to each.\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-3\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>ISTQB test level\u003C/th>\n\u003Cth>Focus\u003C/th>\n\u003Cth>Usual automation level\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Component testing (unit testing)\u003C/td>\n\u003Ctd>Components in isolation\u003C/td>\n\u003Ctd>Unit tests\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Component integration testing\u003C/td>\n\u003Ctd>Interfaces and interactions between components\u003C/td>\n\u003Ctd>Integration tests\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>System testing\u003C/td>\n\u003Ctd>Behavior and capabilities of the whole system\u003C/td>\n\u003Ctd>End-to-end tests\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>System integration testing\u003C/td>\n\u003Ctd>Interfaces with other systems and external services\u003C/td>\n\u003Ctd>Integration or end-to-end tests\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Acceptance testing\u003C/td>\n\u003Ctd>Validation and readiness for deployment\u003C/td>\n\u003Ctd>Any level\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>A test level describes an activity of the test process: who performs it, against which basis, with which objectives. An automation level describes a technical property of one automated test: what it starts and what it replaces with test doubles. Component testing maps directly to unit tests. An acceptance test can be automated at any level: Vocke notes that a check of a feature from the user's point of view does not have to run through the user interface.\u003C/p>\n\u003Ch3 id=\"unit-tests\" class=\"wiki-heading\">Unit tests\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"unit-tests\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>A unit test calls a function, a class or a module directly, in the process of the test framework. Slow or uncontrolled dependencies, such as a database, the network or the clock, are replaced with test doubles. The syllabus notes that component testing often requires test harnesses or unit test frameworks and is normally performed by developers in their development environments.\u003C/p>\n\u003Cp>A unit test of the rule "an expired card is rejected" passes a card with a past expiry date to the validation function and checks the returned error. It needs no server, no database and no browser. Its failure points to one function.\u003C/p>\n\u003Ch3 id=\"integration-tests\" class=\"wiki-heading\">Integration tests\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"integration-tests\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>The term "integration test" has at least two meanings, as Fowler notes. A \u003Cstrong>narrow integration test\u003C/strong> exercises only the part of a service that talks to another component: a database, a message broker, another service. Its scope stays close to that of a unit test. A \u003Cstrong>broad integration test\u003C/strong> needs live versions of all services and a \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-environment\">test environment\u003C/span> with network access.\u003C/p>\n\u003Cp>The ISTQB levels follow a similar split. Component integration testing depends heavily on the integration strategy: bottom-up, top-down or big-bang. System integration testing requires a test environment that is preferably similar to the operational environment.\u003C/p>\n\u003Cp>An integration test of the payment rule sends a request with an expired card to the payment service running with a real database, and checks the HTTP status and the error code in the response. A failure points to the service, its database schema or the mapping between them.\u003C/p>\n\u003Ch3 id=\"end-to-end-tests\" class=\"wiki-heading\">End-to-end tests\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"end-to-end-tests\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>An end-to-end \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-run\">test runs\u003C/span> against the deployed system and goes through an external interface: the user interface or the public API. In the payment example, the test opens the shop in a browser, logs in, adds goods to the cart, pays with an expired card and checks the message on the screen.\u003C/p>\n\u003Cp>Such a test is the closest to how the product is used. It also depends on the most things at once: the build, the configuration, the test environment, the test data, the network and the browser. How end-to-end tests find elements on a page is described in the article \u003Ca href=\"/en/wiki/test-automation/ui-test-locators\" class=\"wiki-link wiki-link--article\">Locators in UI tests\u003C/a>.\u003C/p>\n\u003Ch2 id=\"cost-and-speed-of-each-level\" class=\"wiki-heading\">Cost and speed of each level\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"cost-and-speed-of-each-level\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The syllabus states the general rule: the higher the layer, the lower the test granularity, test isolation and test execution time. Tests in the bottom layer are small, isolated and fast. Tests in the top layer are generally slower. The table compares the levels by the factors that make up the cost of a test.\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-4\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Factor\u003C/th>\n\u003Cth>Unit test\u003C/th>\n\u003Cth>Integration test\u003C/th>\n\u003Cth>End-to-end test\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>What runs\u003C/td>\n\u003Ctd>Code in the test process\u003C/td>\n\u003Ctd>A service with its real dependencies\u003C/td>\n\u003Ctd>The whole deployed system\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Slowest operation\u003C/td>\n\u003Ctd>Function calls in memory\u003C/td>\n\u003Ctd>Database queries, network calls\u003C/td>\n\u003Ctd>Page loads, rendering, all services\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Environment needed\u003C/td>\n\u003Ctd>None\u003C/td>\n\u003Ctd>Database, broker, service containers\u003C/td>\n\u003Ctd>Full test environment\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Failure localization\u003C/td>\n\u003Ctd>One function or class\u003C/td>\n\u003Ctd>One service or interface\u003C/td>\n\u003Ctd>Anywhere in the system\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Reasons to fail without a defect\u003C/td>\n\u003Ctd>Few\u003C/td>\n\u003Ctd>Environment, test data\u003C/td>\n\u003Ctd>Environment, data, timing, interface changes\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>What it cannot find\u003C/td>\n\u003Ctd>Defects in interactions\u003C/td>\n\u003Ctd>Defects in the complete user flow\u003C/td>\n\u003Ctd>Little; it is the closest to real use\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>Exact numbers depend on the product and the infrastructure; there are no universal figures. The difference comes from the operations themselves. A function call in memory takes far less time than a database query over the network, and a query takes less than loading a page that triggers many queries.\u003C/p>\n\u003Cp>Some build tools encode this difference explicitly. In Bazel a test target has a \u003Ccode>size\u003C/code> attribute with the values \u003Ccode>small\u003C/code>, \u003Ccode>medium\u003C/code>, \u003Ccode>large\u003C/code> and \u003Ccode>enormous\u003C/code>. The size sets the default timeout: 1 minute, 5 minutes, 15 minutes and 60 minutes respectively, and the resources reserved for local runs.\u003C/p>\n\u003Cp>Execution time is only part of the cost. A failure of an end-to-end test requires an investigation: the product, the environment, the data or the test itself can be at fault. Tests that fail without a defect in the product are described in the article \u003Ca href=\"/en/wiki/test-automation/flaky-tests\" class=\"wiki-link wiki-link--article\">Flaky tests\u003C/a>.\u003C/p>\n\u003Ch2 id=\"the-test-pyramid\" class=\"wiki-heading\">The test pyramid\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"the-test-pyramid\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The syllabus describes the pyramid as a model showing that tests may have different granularity. The layers represent groups of tests. Tests in the bottom layer check a small piece of functionality each, so many of them are needed to achieve reasonable coverage. Tests in the top layer check a large piece of functionality each, so a few of them are enough.\u003C/p>\n\u003Cdiv class=\"wiki-code wiki-code--plain\">\u003Cpre>\u003Ccode>            /\\\n           /  \\        End-to-end tests: few, slow, broad\n          /----\\\n         /      \\      Integration tests\n        /--------\\\n       /          \\    Unit tests: many, fast, isolated\n      +------------+\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>The syllabus notes that the number and names of the layers differ. Cohn's original pyramid has three layers: unit tests, service tests and UI tests. Another common variant has unit (component) tests, integration (component integration) tests and end-to-end tests.\u003C/p>\n\u003Cp>The middle layer is the least obvious. Cohn defines a service as something the application does in response to some input or set of inputs, and service tests check these services without the user interface. Fowler calls such tests subcutaneous: they act through the service layer or the API of the application just under the user interface. They give many of the benefits of end-to-end tests without the complexity of the user interface.\u003C/p>\n\u003Cp>Vocke warns against taking the names of the layers too literally: the pyramid is simplistic and can be misleading. The shape does not prescribe exact proportions.\u003C/p>\n\u003Ch3 id=\"the-ice-cream-cone-anti-pattern\" class=\"wiki-heading\">The ice cream cone anti-pattern\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"the-ice-cream-cone-anti-pattern\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>The \u003Cstrong>ice cream cone\u003C/strong> is the inverted test pyramid: most checks are manual tests and tests through the user interface, and few are unit tests. Alister Scott described it in his blog WatirMelon in January 2012. He wrote that organizations often create inverted testing pyramids. His own pyramid has a cloud of manual session-based tests on top, and when it is inverted, the shape becomes an ice cream cone.\u003C/p>\n\u003Cp>Fowler's TestPyramid entry describes how such a suite appears. A team automates tests with record and playback tools: a tester records actions in the user interface, and the tool replays them and compares the results. Fowler notes that this works well at first. The suite then grows at the top level and breaks with changes to the interface. Fowler calls the result an ice cream cone.\u003C/p>\n\u003Cp>The anti-pattern describes the proportion, not the existence of end-to-end tests. A suite with a few end-to-end tests for the main user flows on top of many lower-level tests is still a pyramid.\u003C/p>\n\u003Ch2 id=\"other-shapes\" class=\"wiki-heading\">Other shapes\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"other-shapes\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The \u003Cstrong>testing trophy\u003C/strong> was shared by Kent C. Dodds on 6 February 2018 and described in his article of 3 June 2021. It has four layers, from bottom to top: static checks (type checkers and linters), unit tests, integration tests and end-to-end tests. The widest layer is integration. Dodds takes the idea from Guillermo Rauch's advice of 10 December 2016: "Write tests. Not too many. Mostly integration."\u003C/p>\n\u003Cp>The table compares the two shapes.\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-3\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>\u003C/th>\n\u003Cth>Test pyramid\u003C/th>\n\u003Cth>Testing trophy\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Author, year\u003C/td>\n\u003Ctd>Mike Cohn, 2009\u003C/td>\n\u003Ctd>Kent C. Dodds, 2018\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Layers\u003C/td>\n\u003Ctd>Unit, service, UI (original)\u003C/td>\n\u003Ctd>Static, unit, integration, end-to-end\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Widest layer\u003C/td>\n\u003Ctd>Unit tests\u003C/td>\n\u003Ctd>Integration tests\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Argument\u003C/td>\n\u003Ctd>Lower tests are faster and cheaper\u003C/td>\n\u003Ctd>Return on investment: confidence for time spent\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Stated scope\u003C/td>\n\u003Ctd>General\u003C/td>\n\u003Ctd>One codebase; Dodds applied it to monoliths\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>Both shapes put few end-to-end tests at the top, and both relate the number of tests to their cost. They differ in where most of the effort goes. Dodds explains his support for Rauch's advice by his experience as a user interface engineer, and states that the classification itself matters less than the goal.\u003C/p>\n\u003Cp>In both models the meaning of "integration" has to be agreed within the team. Dodds defines integration tests as tests of several units working together, which is closer to Fowler's narrow integration tests than to broad ones.\u003C/p>\n\u003Ch2 id=\"choosing-the-level-for-a-check\" class=\"wiki-heading\">Choosing the level for a check\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"choosing-the-level-for-a-check\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The usual rule follows from the pyramid: a check belongs to the lowest level at which it can find the defect it is aimed at. Vocke formulates two consequences. If a higher-level test finds an error and no lower-level test fails, a lower-level test is missing and should be written. And a high-level test that only repeats checks already made at lower levels adds cost without adding confidence.\u003C/p>\n\u003Cp>A check needs a higher level when its defect exists only there. Wiring, configuration, the database schema and the behavior of a real browser cannot be checked by a unit test, because it replaces exactly these parts with test doubles.\u003C/p>\n\u003Cp>The table distributes the checks of one feature, a promo code at checkout, between the levels.\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-3\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Check\u003C/th>\n\u003Cth>Level\u003C/th>\n\u003Cth>Why this level\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Promo code format is validated\u003C/td>\n\u003Ctd>Unit\u003C/td>\n\u003Ctd>Pure function of the input\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Discount at the boundaries of the order amount\u003C/td>\n\u003Ctd>Unit\u003C/td>\n\u003Ctd>Many input values, each test is cheap\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Expired promo code is rejected\u003C/td>\n\u003Ctd>Unit\u003C/td>\n\u003Ctd>The clock is replaced with a test double\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Order with a discount is saved and read back\u003C/td>\n\u003Ctd>Integration\u003C/td>\n\u003Ctd>Depends on the real database schema\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Request for a discount follows the agreed API\u003C/td>\n\u003Ctd>Contract test\u003C/td>\n\u003Ctd>Two services, each tested separately\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>User applies a code and sees the new total\u003C/td>\n\u003Ctd>End-to-end\u003C/td>\n\u003Ctd>Whole flow through the browser, one test\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>The values for the second row are chosen by test design, described in the article \u003Ca href=\"/en/wiki/test-design/equivalence-partitioning-and-boundary-values\" class=\"wiki-link wiki-link--article\">Equivalence partitioning and boundary values\u003C/a>. The pyramid only says that such numerous checks should be cheap.\u003C/p>\n\u003Ch2 id=\"contract-tests-between-services\" class=\"wiki-heading\">Contract tests between services\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"contract-tests-between-services\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The interaction of two services can be checked without deploying them together. A \u003Cstrong>contract test\u003C/strong> checks each side of an interface separately against a shared description of the requests and responses. In \u003Cstrong>consumer-driven contracts\u003C/strong>, a pattern described by Ian Robinson in 2006, consumers state their expectations of a provider, and the provider's obligations are the sum of these expectations.\u003C/p>\n\u003Cp>Pact implements this pattern. The consumer's tests send real requests to a mock provider created by Pact. The mock checks each request and returns the expected response. When the consumer tests pass, Pact writes a pact file that describes each interaction. In provider verification, Pact replays every recorded request against the real provider and compares its response with the minimal expected response.\u003C/p>\n\u003Cdiv class=\"wiki-code wiki-code--plain\">\u003Cpre>\u003Ccode>Consumer tests                        Provider verification\n |                                     |\n +-- request --> Pact mock provider    +-- recorded request --> real provider\n |               |                     |                         |\n |               +-- expected response |                         +-- actual response\n |                                     |                               |\n +--> pact file -----------------------+--> compared with expected <---+\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>Only the parts of the provider that consumers use are checked, so the provider can change unused behavior freely. Contract tests take over part of the work of broad integration tests without a full test environment. They do not check the deployed system, so a few end-to-end tests remain.\u003C/p>\n\u003Ch2 id=\"levels-coverage-and-continuous-integration\" class=\"wiki-heading\">Levels, coverage and continuous integration\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"levels-coverage-and-continuous-integration\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>Code coverage is measured mostly for unit and integration tests, whose process can be instrumented. \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"requirements-coverage\">Requirements coverage\u003C/span> is computed from links between requirements and tests and is typical for system and acceptance tests. The two measures are compared in the article \u003Ca href=\"/en/wiki/test-management/requirements-traceability\" class=\"wiki-link wiki-link--article\">Requirements traceability and coverage\u003C/a>.\u003C/p>\n\u003Cp>The level of a test also determines where it can run in a \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"continuous-integration\">continuous integration\u003C/span> (CI) \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"ci-pipeline\">pipeline\u003C/span>. Unit tests need no environment and can run on every change. Integration tests need their dependencies, for example started in containers for the duration of the job. End-to-end tests need a deployed system. How tests are triggered by pipeline events, run in parallel and reported is described in the article \u003Ca href=\"/en/wiki/test-automation/automated-tests-in-ci\" class=\"wiki-link wiki-link--article\">Automated tests in CI\u003C/a>.\u003C/p>\n\u003Ch2 id=\"terms\" class=\"wiki-heading\">Terms\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"terms\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cdiv class=\"wiki-table wiki-table--cols-2\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Term\u003C/th>\n\u003Cth>Meaning\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Test automation level\u003C/td>\n\u003Ctd>Group of automated tests that exercise a part of the system of similar size\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Unit test\u003C/td>\n\u003Ctd>Automated test that checks one component in isolation, in the test process\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Integration test\u003C/td>\n\u003Ctd>Automated test that checks the interaction of components or of a system with other systems\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>End-to-end test\u003C/td>\n\u003Ctd>Automated test that checks the whole deployed system through its user interface or public API\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Test pyramid\u003C/td>\n\u003Ctd>Model of many small, fast tests at the bottom and few large, slow tests at the top\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Ice cream cone\u003C/td>\n\u003Ctd>Anti-pattern of an inverted pyramid: mostly manual and UI tests, few unit tests\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Testing trophy\u003C/td>\n\u003Ctd>Model of Kent C. Dodds with static, unit, integration and end-to-end layers, integration the widest\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Contract test\u003C/td>\n\u003Ctd>Test that checks each side of an interface separately against a shared description of requests and responses\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Consumer-driven contract\u003C/td>\n\u003Ctd>Contract of a provider formed from the expectations of its consumers\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Ch2 id=\"sources\" class=\"wiki-heading\">Sources\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"sources\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cul>\n\u003Cli>\u003Ca href=\"https://www.istqb.org/wp-content/uploads/2024/11/ISTQB_CTFL_Syllabus_v4.0.1.pdf\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">ISTQB Certified Tester Foundation Level Syllabus v4.0.1, ISTQB, 2024\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://istqb.org/istqb-releases-certified-tester-foundation-level-v4-0-ctfl/\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">ISTQB releases Certified Tester Foundation Level v4.0, ISTQB, 2023\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://www.pearson.com/en-us/subject-catalog/p/succeeding-with-agile-software-development-using-scrum/P200000009360/9780321660565\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Succeeding with Agile: Software Development Using Scrum, Mike Cohn, Addison-Wesley, 2009\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">The Forgotten Layer of the Test Automation Pyramid, Mike Cohn\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://martinfowler.com/bliki/TestPyramid.html\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">TestPyramid, Martin Fowler, 2012\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://martinfowler.com/bliki/IntegrationTest.html\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">IntegrationTest, Martin Fowler, 2018\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://martinfowler.com/articles/practical-test-pyramid.html\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">The Practical Test Pyramid, Ham Vocke, 2018\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://watirmelon.blog/2012/01/31/introducing-the-software-testing-ice-cream-cone/\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Introducing the software testing ice-cream cone (anti-pattern), Alister Scott, 2012\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://kentcdodds.com/blog/the-testing-trophy-and-testing-classifications\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">The Testing Trophy and Testing Classifications, Kent C. Dodds, 2021\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://martinfowler.com/articles/consumerDrivenContracts.html\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Consumer-Driven Contracts: A Service Evolution Pattern, Ian Robinson, 2006\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://docs.pact.io/\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Introduction to Pact, Pact documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://docs.pact.io/getting_started/how_pact_works\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">How Pact works, Pact documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://bazel.build/reference/be/common-definitions\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Common definitions: attributes common to all test rules, Bazel documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003C/ul>\n",[10,14,17,20,24,27,30,33,36,39,42,45,48,51,54],{"id":11,"level":12,"text":13},"the-problem-the-test-pyramid-solves",2,"The problem the test pyramid solves",{"id":15,"level":12,"text":16},"history","History",{"id":18,"level":12,"text":19},"test-levels-and-automation-levels","Test levels and automation levels",{"id":21,"level":22,"text":23},"unit-tests",3,"Unit tests",{"id":25,"level":22,"text":26},"integration-tests","Integration tests",{"id":28,"level":22,"text":29},"end-to-end-tests","End-to-end tests",{"id":31,"level":12,"text":32},"cost-and-speed-of-each-level","Cost and speed of each level",{"id":34,"level":12,"text":35},"the-test-pyramid","The test pyramid",{"id":37,"level":22,"text":38},"the-ice-cream-cone-anti-pattern","The ice cream cone anti-pattern",{"id":40,"level":12,"text":41},"other-shapes","Other shapes",{"id":43,"level":12,"text":44},"choosing-the-level-for-a-check","Choosing the level for a check",{"id":46,"level":12,"text":47},"contract-tests-between-services","Contract tests between services",{"id":49,"level":12,"text":50},"levels-coverage-and-continuous-integration","Levels, coverage and continuous integration",{"id":52,"level":12,"text":53},"terms","Terms",{"id":55,"level":12,"text":56},"sources","Sources",2691,1793718928916]