[{"data":1,"prerenderedAt":42},["ShallowReactive",2],{"wiki-test-management-test-metrics-en":3},{"meta":4,"html":8,"headings":9,"words":41},{"title":5,"description":6,"published":7,"updated":7},"Test metrics","Test metrics: pass rate, defect density, defects found in production, time to fix. How each is computed, where it misleads, ISTQB categories.","2026-11-03","\u003Cp>A \u003Cstrong>test metric\u003C/strong> is a measure that quantifies an attribute of testing or of the product under test: progress of execution, number and \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"severity\">severity\u003C/span> of \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"defect\">defects\u003C/span>, coverage, cost. Metrics are computed from the data that testing produces (results of \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-run\">test runs\u003C/span>, \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"defect-report\">defect reports\u003C/span>, links to requirements) and are used to monitor testing, to check \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"exit-criteria\">exit criteria\u003C/span> and to support decisions about a release.\u003C/p>\n\u003Cp>The ISTQB Foundation Level syllabus (version 4.0.1, section 5.3.1) groups test metrics into categories and names typical examples in each. ISO/IEC/IEEE 29119-3, the international standard for test documentation (current edition 2021), includes measures as a part of the test status report and the test completion report, but does not prescribe which measures to use. The choice of metrics, their definitions and their thresholds are set by the organization and the project.\u003C/p>\n\u003Ch2 id=\"what-test-metrics-are-for\" class=\"wiki-heading\">What test metrics are for\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"what-test-metrics-are-for\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>Testing produces large amounts of data: hundreds of results per run, dozens of defects per release. A metric compresses this data into a number that can be compared: with the plan, with the exit criteria, with the previous release.\u003C/p>\n\u003Cp>The ISTQB syllabus connects metrics with test monitoring and control. Monitoring compares the actual state of testing with the plan, control takes actions when they differ. Metrics are the means of comparison: "120 of 300 planned \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-case\">test cases\u003C/span> executed with three days left" is a fact on which a decision can be based, "testing is going well" is not.\u003C/p>\n\u003Cp>A metric answers a specific question. The questions most often asked during testing:\u003C/p>\n\u003Cul>\n\u003Cli>how far has the execution of planned tests progressed;\u003C/li>\n\u003Cli>what is the quality of the product: how many defects there are and how severe;\u003C/li>\n\u003Cli>how effective is testing: how many defects it finds before release;\u003C/li>\n\u003Cli>how fast are defects resolved;\u003C/li>\n\u003Cli>what is covered by tests and what is not.\u003C/li>\n\u003C/ul>\n\u003Ch2 id=\"categories-of-metrics\" class=\"wiki-heading\">Categories of metrics\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"categories-of-metrics\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The ISTQB syllabus names seven categories of metrics:\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-2\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Category\u003C/th>\n\u003Cth>Examples\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Project progress\u003C/td>\n\u003Ctd>Task completion, resource usage, test effort\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Test progress\u003C/td>\n\u003Ctd>Progress of test case implementation, test environment preparation, number of test cases run and not run, passed and failed, test execution time\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Product quality\u003C/td>\n\u003Ctd>Availability, response time, mean time to failure\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Defects\u003C/td>\n\u003Ctd>Number and priorities of defects found and fixed, defect density, defect detection percentage\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Risk\u003C/td>\n\u003Ctd>Residual risk level\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Coverage\u003C/td>\n\u003Ctd>Requirements coverage, code coverage\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Cost\u003C/td>\n\u003Ctd>Cost of testing, organizational cost of quality\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>The sections below describe four metrics that \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-management-system\">test management systems\u003C/span> compute most often, with their formulas and the ways in which they mislead. Coverage metrics are described in the article \u003Ca href=\"/en/wiki/test-management/requirements-traceability\" class=\"wiki-link wiki-link--article\">Requirements traceability and coverage\u003C/a>.\u003C/p>\n\u003Ch2 id=\"pass-rate\" class=\"wiki-heading\">Pass rate\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"pass-rate\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The \u003Cstrong>pass rate\u003C/strong> is the share of passed test cases. It is computed for a test run, for a set of runs or for a release:\u003C/p>\n\u003Cdiv class=\"wiki-code wiki-code--plain\">\u003Cpre>\u003Ccode>pass rate = passed / denominator x 100%\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>The value depends on the denominator, and three are in use: all planned test cases of the run, executed test cases, or test cases that reached a verdict (passed or failed). For a run with 200 planned test cases, of which 150 passed, 10 failed, 15 are blocked and 25 not yet executed:\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-3\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Denominator\u003C/th>\n\u003Cth>Calculation\u003C/th>\n\u003Cth>Pass rate\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Planned\u003C/td>\n\u003Ctd>150 / 200\u003C/td>\n\u003Ctd>75 percent\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Executed (passed, failed, blocked)\u003C/td>\n\u003Ctd>150 / 175\u003C/td>\n\u003Ctd>86 percent\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>With a verdict (passed, failed)\u003C/td>\n\u003Ctd>150 / 160\u003C/td>\n\u003Ctd>94 percent\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>The same run gives 75, 86 or 94 percent. A pass rate without its denominator cannot be interpreted.\u003C/p>\n\u003Cp>Where the pass rate misleads:\u003C/p>\n\u003Cul>\n\u003Cli>\u003Cstrong>Unequal weight.\u003C/strong> All test cases count the same. A run where 199 cosmetic checks pass and the payment check fails has a pass rate of 99.5 percent.\u003C/li>\n\u003Cli>\u003Cstrong>Dependence on the number of tests.\u003C/strong> Adding many simple test cases that always pass raises the rate without changing the product.\u003C/li>\n\u003Cli>\u003Cstrong>Unexecuted tests.\u003C/strong> If blocked and unexecuted test cases are left out of the denominator, a run where half of the tests could not be executed can still show a high pass rate.\u003C/li>\n\u003Cli>\u003Cstrong>Repeated executions.\u003C/strong> If automated tests are retried until they pass and only the last attempt is counted, the rate hides unstable tests and \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"intermittent-defect\">intermittent defects\u003C/span>.\u003C/li>\n\u003C/ul>\n\u003Cp>The pass rate is informative together with the number of failed and blocked test cases and with the list of failed test cases of high priority.\u003C/p>\n\u003Ch2 id=\"defect-density\" class=\"wiki-heading\">Defect density\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"defect-density\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>\u003Cstrong>Defect density\u003C/strong> is, in the wording of the ISTQB glossary, the number of defects per unit size of a work product:\u003C/p>\n\u003Cdiv class=\"wiki-code wiki-code--plain\">\u003Cpre>\u003Ccode>defect density = number of defects / size\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>The size is measured in units that suit the work product: thousands of lines of code (KLOC), function points, number of requirements, pages of a specification. For two components:\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-4\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Component\u003C/th>\n\u003Cth>Defects\u003C/th>\n\u003Cth>Size\u003C/th>\n\u003Cth>Density\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Payment\u003C/td>\n\u003Ctd>24\u003C/td>\n\u003Ctd>8 KLOC\u003C/td>\n\u003Ctd>3.0 defects per KLOC\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Catalog\u003C/td>\n\u003Ctd>18\u003C/td>\n\u003Ctd>12 KLOC\u003C/td>\n\u003Ctd>1.5 defects per KLOC\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>The payment component has more defects per unit of size, which points to it as a candidate for additional testing or review.\u003C/p>\n\u003Cp>Where defect density misleads:\u003C/p>\n\u003Cul>\n\u003Cli>\u003Cstrong>It counts found defects, not existing ones.\u003C/strong> A component that was tested less has fewer found defects and a lower density. Low density can mean high quality or little testing.\u003C/li>\n\u003Cli>\u003Cstrong>It depends on counting rules.\u003C/strong> Whether rejected reports, duplicates and defects of all severities are counted changes the number. The rules are fixed before comparing.\u003C/li>\n\u003Cli>\u003Cstrong>It depends on the size measure.\u003C/strong> Lines of code differ between languages and coding styles; the same functionality can be 2 KLOC in one language and 6 KLOC in another.\u003C/li>\n\u003Cli>\u003Cstrong>It ignores severity.\u003C/strong> Ten cosmetic defects and ten defects that lose data give the same density.\u003C/li>\n\u003C/ul>\n\u003Cp>Defect density is used to compare components of one product measured in the same way, and to compare a component with itself over releases. Comparisons between products, teams or languages are unreliable for the reasons above.\u003C/p>\n\u003Ch2 id=\"defects-found-in-production\" class=\"wiki-heading\">Defects found in production\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"defects-found-in-production\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The share of defects that testing did not find before release is measured by the \u003Cstrong>defect detection percentage\u003C/strong> (DDP). The ISTQB glossary defines it as the number of defects found by a test level, divided by the number found by that test level and any other means afterwards:\u003C/p>\n\u003Cdiv class=\"wiki-code wiki-code--plain\">\u003Cpre>\u003Ccode>DDP = defects found by testing / (defects found by testing + defects found later) x 100%\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>The complementary value, the share of defects found after release, is often called defect leakage or escaped defects. For a release in which testing found 85 defects and users reported 15 more in the first 90 days:\u003C/p>\n\u003Cdiv class=\"wiki-code wiki-code--plain\">\u003Cpre>\u003Ccode>DDP = 85 / (85 + 15) x 100% = 85%\ndefects found in production = 15 / 100 x 100% = 15%\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>Where it misleads:\u003C/p>\n\u003Cul>\n\u003Cli>\u003Cstrong>The value changes over time.\u003C/strong> Defects found after release keep arriving for months. A DDP computed a week after release is higher than the same DDP computed half a year later. The window ("first 90 days") is part of the definition.\u003C/li>\n\u003Cli>\u003Cstrong>It needs a definition of "the same" defect.\u003C/strong> A user report that turns out to be a known defect, deferred before the release, is not an escaped defect. Without this rule, deliberate decisions are counted as testing failures.\u003C/li>\n\u003Cli>\u003Cstrong>Severity matters more than number.\u003C/strong> Fifteen cosmetic defects in production and one defect that charges customers twice give very different pictures of a release. DDP is often computed separately for severe defects.\u003C/li>\n\u003Cli>\u003Cstrong>It depends on how users report.\u003C/strong> A product with few active users or without a channel for reports shows few production defects regardless of its quality.\u003C/li>\n\u003C/ul>\n\u003Cp>DDP measures testing as a whole, not individual testers. It also depends on the quality of the code that reached testing and on what was in scope for testing.\u003C/p>\n\u003Ch2 id=\"time-to-fix\" class=\"wiki-heading\">Time to fix\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"time-to-fix\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The \u003Cstrong>time to fix\u003C/strong> a defect is the time from the moment it is reported to the moment it is resolved. It is computed for a set of defects as the mean or the median. What "resolved" means is defined explicitly: the fix is merged, the fix is confirmed by testing, or the report is closed.\u003C/p>\n\u003Cp>The mean and the median differ when there are outliers. For ten defects fixed in 1, 1, 2, 2, 2, 3, 3, 4, 5 and 47 days:\u003C/p>\n\u003Cdiv class=\"wiki-code wiki-code--plain\">\u003Cpre>\u003Ccode>mean   = (1 + 1 + 2 + 2 + 2 + 3 + 3 + 4 + 5 + 47) / 10 = 7 days\nmedian = (2 + 3) / 2 = 2.5 days\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>One defect that waited 47 days moves the mean from about 2.6 days (for the other nine) to 7 days. The median describes the typical case, the mean is sensitive to long waits, and both are informative together with the list of defects that took longest.\u003C/p>\n\u003Cp>Where it misleads:\u003C/p>\n\u003Cul>\n\u003Cli>\u003Cstrong>Deferred and reopened defects.\u003C/strong> A defect deliberately deferred for a later release accumulates time without anyone working on it. A reopened defect can be counted from the first report or from the reopening. The rules are fixed in advance.\u003C/li>\n\u003Cli>\u003Cstrong>Severity.\u003C/strong> Critical defects are usually fixed first. A mean over all severities mixes very different queues. The metric is computed per severity or per priority.\u003C/li>\n\u003Cli>\u003Cstrong>Splitting.\u003C/strong> Closing a report and opening a new one for the remaining part of the problem shortens the measured time without speeding up the fix.\u003C/li>\n\u003C/ul>\n\u003Cp>The time to fix a defect found in testing is a different metric from the time to recover from a failure in production. The DevOps Research and Assessment (DORA) program measures the latter as failed deployment recovery time; since 2023 this name replaces the earlier "mean time to recover".\u003C/p>\n\u003Ch2 id=\"how-metrics-mislead\" class=\"wiki-heading\">How metrics mislead\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"how-metrics-mislead\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The examples above share common patterns.\u003C/p>\n\u003Cp>\u003Cstrong>A metric becomes a target.\u003C/strong> The economist Charles Goodhart described in 1975 how a statistical indicator stops being reliable once it is used as a target of control. The anthropologist Marilyn Strathern generalized this observation in 1997 to measures in general: a measure that becomes a target stops being a good measure. In testing, a target "pass rate above 95 percent" encourages removing failing tests from runs; a target "few defects in production" encourages classifying production defects as change requests.\u003C/p>\n\u003Cp>\u003Cstrong>One metric without others.\u003C/strong> Each metric describes one side of the situation. A high pass rate with many blocked tests, a low defect density with little testing, a short time to fix with many reopened defects are each misleading on their own. Metrics are read in sets: progress together with quality, quantity together with severity.\u003C/p>\n\u003Cp>\u003Cstrong>Comparison without equal definitions.\u003C/strong> Two teams that count defects, test cases or time differently produce numbers that cannot be compared. Before comparing, the definitions are aligned: what is counted, in which window, with which denominator.\u003C/p>\n\u003Cp>\u003Cstrong>Absolute values instead of trends.\u003C/strong> A defect density of 3 per KLOC says little by itself. The same component going from 1.5 to 3 between releases says more. Many metrics are informative as a series over releases rather than as a single value.\u003C/p>\n\u003Ch2 id=\"test-metrics-and-delivery-metrics\" class=\"wiki-heading\">Test metrics and delivery metrics\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"test-metrics-and-delivery-metrics\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>Test metrics describe testing and the product found by testing. Delivery metrics describe how changes reach production. The DORA program, which studies software delivery performance, uses five such metrics since 2024: change lead time, deployment frequency, failed deployment recovery time, change fail rate and deployment rework rate.\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-3\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>\u003C/th>\n\u003Cth>Test metrics\u003C/th>\n\u003Cth>DORA delivery metrics\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Object\u003C/td>\n\u003Ctd>Testing and the product before release\u003C/td>\n\u003Ctd>The process of delivering changes to production\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Data\u003C/td>\n\u003Ctd>Test results, defect reports, links to requirements\u003C/td>\n\u003Ctd>Commits, deployments, incidents\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Examples\u003C/td>\n\u003Ctd>Pass rate, defect density, DDP, time to fix\u003C/td>\n\u003Ctd>Change lead time, deployment frequency, change fail rate\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Typical question\u003C/td>\n\u003Ctd>Is the release ready? Where are the risks?\u003C/td>\n\u003Ctd>How fast and how reliably do changes reach users?\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>The two sets complement each other. A high change fail rate in delivery metrics points to defects that testing let through, and DDP and the list of escaped defects show which ones. A long change lead time can be caused by long test runs, and test progress metrics show where the time goes. How automated tests run in the delivery \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"ci-pipeline\">pipeline\u003C/span> is described in the article \u003Ca href=\"/en/wiki/test-automation/automated-tests-in-ci\" class=\"wiki-link wiki-link--article\">Automated tests in CI\u003C/a>.\u003C/p>\n\u003Ch2 id=\"terms\" class=\"wiki-heading\">Terms\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"terms\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cdiv class=\"wiki-table wiki-table--cols-2\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Term\u003C/th>\n\u003Cth>Meaning\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Test metric\u003C/td>\n\u003Ctd>Measure that quantifies an attribute of testing or of the product under test\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Pass rate\u003C/td>\n\u003Ctd>Share of passed test cases relative to a stated denominator\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Defect density\u003C/td>\n\u003Ctd>Number of defects per unit size of a work product\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Defect detection percentage (DDP)\u003C/td>\n\u003Ctd>Number of defects found by a test level, divided by the number found by that level and any other means afterwards\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Defects found in production\u003C/td>\n\u003Ctd>Share of defects found after release; the complement of DDP\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Time to fix\u003C/td>\n\u003Ctd>Time from the report of a defect to its resolution, as a mean or a median\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>DORA\u003C/td>\n\u003Ctd>DevOps Research and Assessment: research program that defines metrics of software delivery performance\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Goodhart's law\u003C/td>\n\u003Ctd>Observation that a measure used as a target stops being a reliable measure\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Ch2 id=\"sources\" class=\"wiki-heading\">Sources\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"sources\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cul>\n\u003Cli>\u003Ca href=\"https://www.istqb.org/wp-content/uploads/2024/11/ISTQB_CTFL_Syllabus_v4.0.1.pdf\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">ISTQB Certified Tester Foundation Level Syllabus v4.0.1, ISTQB, 2024\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://glossary.istqb.org/\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">ISTQB Glossary\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://www.iso.org/standard/79429.html\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">ISO/IEC/IEEE 29119-3:2021, Software testing, Part 3: Test documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://dora.dev/guides/dora-metrics/history/\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">A history of DORA's software delivery metrics, DORA\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://www.cambridge.org/core/journals/european-review/article/improving-ratings-audit-in-the-british-university-system/FC2EE640C0C44E3DB87C29FB666E9AAB\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">"Improving ratings": audit in the British University system, M. Strathern, European Review, 1997\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003C/ul>\n",[10,14,17,20,23,26,29,32,35,38],{"id":11,"level":12,"text":13},"what-test-metrics-are-for",2,"What test metrics are for",{"id":15,"level":12,"text":16},"categories-of-metrics","Categories of metrics",{"id":18,"level":12,"text":19},"pass-rate","Pass rate",{"id":21,"level":12,"text":22},"defect-density","Defect density",{"id":24,"level":12,"text":25},"defects-found-in-production","Defects found in production",{"id":27,"level":12,"text":28},"time-to-fix","Time to fix",{"id":30,"level":12,"text":31},"how-metrics-mislead","How metrics mislead",{"id":33,"level":12,"text":34},"test-metrics-and-delivery-metrics","Test metrics and delivery metrics",{"id":36,"level":12,"text":37},"terms","Terms",{"id":39,"level":12,"text":40},"sources","Sources",2025,1793718881111]