Test metrics

A test metric is a measure that quantifies an attribute of testing or of the product under test: progress of execution, number and severity of defects, coverage, cost. Metrics are computed from the data that testing produces (results of test runs, defect reports, links to requirements) and are used to monitor testing, to check exit criteria and to support decisions about a release.

The ISTQB Foundation Level syllabus (version 4.0.1, section 5.3.1) groups test metrics into categories and names typical examples in each. ISO/IEC/IEEE 29119-3, the international standard for test documentation (current edition 2021), includes measures as a part of the test status report and the test completion report, but does not prescribe which measures to use. The choice of metrics, their definitions and their thresholds are set by the organization and the project.

What test metrics are for

Testing produces large amounts of data: hundreds of results per run, dozens of defects per release. A metric compresses this data into a number that can be compared: with the plan, with the exit criteria, with the previous release.

The ISTQB syllabus connects metrics with test monitoring and control. Monitoring compares the actual state of testing with the plan, control takes actions when they differ. Metrics are the means of comparison: "120 of 300 planned test cases executed with three days left" is a fact on which a decision can be based, "testing is going well" is not.

A metric answers a specific question. The questions most often asked during testing:

• how far has the execution of planned tests progressed;
• what is the quality of the product: how many defects there are and how severe;
• how effective is testing: how many defects it finds before release;
• how fast are defects resolved;
• what is covered by tests and what is not.

Categories of metrics

The ISTQB syllabus names seven categories of metrics:

The sections below describe four metrics that test management systems compute most often, with their formulas and the ways in which they mislead. Coverage metrics are described in the article Requirements traceability and coverage.

Pass rate

The pass rate is the share of passed test cases. It is computed for a test run, for a set of runs or for a release:

pass rate = passed / denominator x 100%

The value depends on the denominator, and three are in use: all planned test cases of the run, executed test cases, or test cases that reached a verdict (passed or failed). For a run with 200 planned test cases, of which 150 passed, 10 failed, 15 are blocked and 25 not yet executed:

The same run gives 75, 86 or 94 percent. A pass rate without its denominator cannot be interpreted.

Where the pass rate misleads:

• Unequal weight. All test cases count the same. A run where 199 cosmetic checks pass and the payment check fails has a pass rate of 99.5 percent.
• Dependence on the number of tests. Adding many simple test cases that always pass raises the rate without changing the product.
• Unexecuted tests. If blocked and unexecuted test cases are left out of the denominator, a run where half of the tests could not be executed can still show a high pass rate.
• Repeated executions. If automated tests are retried until they pass and only the last attempt is counted, the rate hides unstable tests and intermittent defects.

The pass rate is informative together with the number of failed and blocked test cases and with the list of failed test cases of high priority.

Defect density

Defect density is, in the wording of the ISTQB glossary, the number of defects per unit size of a work product:

defect density = number of defects / size

The size is measured in units that suit the work product: thousands of lines of code (KLOC), function points, number of requirements, pages of a specification. For two components:

The payment component has more defects per unit of size, which points to it as a candidate for additional testing or review.

Where defect density misleads:

• It counts found defects, not existing ones. A component that was tested less has fewer found defects and a lower density. Low density can mean high quality or little testing.
• It depends on counting rules. Whether rejected reports, duplicates and defects of all severities are counted changes the number. The rules are fixed before comparing.
• It depends on the size measure. Lines of code differ between languages and coding styles; the same functionality can be 2 KLOC in one language and 6 KLOC in another.
• It ignores severity. Ten cosmetic defects and ten defects that lose data give the same density.

Defect density is used to compare components of one product measured in the same way, and to compare a component with itself over releases. Comparisons between products, teams or languages are unreliable for the reasons above.

Defects found in production

The share of defects that testing did not find before release is measured by the defect detection percentage (DDP). The ISTQB glossary defines it as the number of defects found by a test level, divided by the number found by that test level and any other means afterwards:

DDP = defects found by testing / (defects found by testing + defects found later) x 100%

The complementary value, the share of defects found after release, is often called defect leakage or escaped defects. For a release in which testing found 85 defects and users reported 15 more in the first 90 days:

DDP = 85 / (85 + 15) x 100% = 85%
defects found in production = 15 / 100 x 100% = 15%

Where it misleads:

• The value changes over time. Defects found after release keep arriving for months. A DDP computed a week after release is higher than the same DDP computed half a year later. The window ("first 90 days") is part of the definition.
• It needs a definition of "the same" defect. A user report that turns out to be a known defect, deferred before the release, is not an escaped defect. Without this rule, deliberate decisions are counted as testing failures.
• Severity matters more than number. Fifteen cosmetic defects in production and one defect that charges customers twice give very different pictures of a release. DDP is often computed separately for severe defects.
• It depends on how users report. A product with few active users or without a channel for reports shows few production defects regardless of its quality.

DDP measures testing as a whole, not individual testers. It also depends on the quality of the code that reached testing and on what was in scope for testing.

Time to fix

The time to fix a defect is the time from the moment it is reported to the moment it is resolved. It is computed for a set of defects as the mean or the median. What "resolved" means is defined explicitly: the fix is merged, the fix is confirmed by testing, or the report is closed.

The mean and the median differ when there are outliers. For ten defects fixed in 1, 1, 2, 2, 2, 3, 3, 4, 5 and 47 days:

mean   = (1 + 1 + 2 + 2 + 2 + 3 + 3 + 4 + 5 + 47) / 10 = 7 days
median = (2 + 3) / 2 = 2.5 days

One defect that waited 47 days moves the mean from about 2.6 days (for the other nine) to 7 days. The median describes the typical case, the mean is sensitive to long waits, and both are informative together with the list of defects that took longest.

Where it misleads:

• Deferred and reopened defects. A defect deliberately deferred for a later release accumulates time without anyone working on it. A reopened defect can be counted from the first report or from the reopening. The rules are fixed in advance.
• Severity. Critical defects are usually fixed first. A mean over all severities mixes very different queues. The metric is computed per severity or per priority.
• Splitting. Closing a report and opening a new one for the remaining part of the problem shortens the measured time without speeding up the fix.

The time to fix a defect found in testing is a different metric from the time to recover from a failure in production. The DevOps Research and Assessment (DORA) program measures the latter as failed deployment recovery time; since 2023 this name replaces the earlier "mean time to recover".

How metrics mislead

The examples above share common patterns.

A metric becomes a target. The economist Charles Goodhart described in 1975 how a statistical indicator stops being reliable once it is used as a target of control. The anthropologist Marilyn Strathern generalized this observation in 1997 to measures in general: a measure that becomes a target stops being a good measure. In testing, a target "pass rate above 95 percent" encourages removing failing tests from runs; a target "few defects in production" encourages classifying production defects as change requests.

One metric without others. Each metric describes one side of the situation. A high pass rate with many blocked tests, a low defect density with little testing, a short time to fix with many reopened defects are each misleading on their own. Metrics are read in sets: progress together with quality, quantity together with severity.

Comparison without equal definitions. Two teams that count defects, test cases or time differently produce numbers that cannot be compared. Before comparing, the definitions are aligned: what is counted, in which window, with which denominator.

Absolute values instead of trends. A defect density of 3 per KLOC says little by itself. The same component going from 1.5 to 3 between releases says more. Many metrics are informative as a series over releases rather than as a single value.

Test metrics and delivery metrics

Test metrics describe testing and the product found by testing. Delivery metrics describe how changes reach production. The DORA program, which studies software delivery performance, uses five such metrics since 2024: change lead time, deployment frequency, failed deployment recovery time, change fail rate and deployment rework rate.

The two sets complement each other. A high change fail rate in delivery metrics points to defects that testing let through, and DDP and the list of escaped defects show which ones. A long change lead time can be caused by long test runs, and test progress metrics show where the time goes. How automated tests run in the delivery pipeline is described in the article Automated tests in CI.

Terms

Sources

• ISTQB Certified Tester Foundation Level Syllabus v4.0.1, ISTQB, 2024(opens in a new tab)
• ISTQB Glossary(opens in a new tab)
• ISO/IEC/IEEE 29119-3:2021, Software testing, Part 3: Test documentation(opens in a new tab)
• A history of DORA's software delivery metrics, DORA(opens in a new tab)
• "Improving ratings": audit in the British University system, M. Strathern, European Review, 1997(opens in a new tab)