Requirements traceability and coverage

Requirements traceability is the ability to follow a requirement from its source through design, implementation and tests to the results of testing, and back. Gotel and Finkelstein defined it in 1994 as the ability to describe and follow the life of a requirement in both a forwards and backwards direction. The vocabulary standard ISO/IEC/IEEE 24765 defines traceability in general as the degree to which a relationship can be established between two or more products of the development process.

In testing, traceability means links between the test basis (requirements, user stories, risks) and testware (test cases, test results, defects). The ISTQB Foundation Level syllabus (version 4.0.1, section 1.4.4) treats establishing and maintaining such traceability throughout the test process as a condition of effective test monitoring and control. Traceability is the basis of two figures that a test management system computes: requirements coverage and the set of tests affected by a change.

The problem traceability solves

A set of requirements and a set of test cases can exist side by side without any link between them. Each of them can be complete in itself, and the following questions still have no answer:

which requirements have no test at all;
which requirements were checked in this release and with what result;
which test cases have to be executed again when a requirement changes;
why a test case exists and whether it is still needed;
which requirements are affected by the open defects of a release.

The ISTQB syllabus names the uses of traceability in testing: evaluating coverage, determining the impact of changes, supporting audits and IT governance, and making test progress and completion reports easier to understand, because they can show the status of the elements of the test basis: requirements that passed their tests, failed them or have tests pending.

Directions of traceability

Traceability has two directions.

Forward traceability goes from a requirement to the work products derived from it: design, code, test cases, test results. It answers the question "is this requirement implemented and tested?".
Backward traceability goes from a work product back to the requirement it comes from. It answers the question "why does this test case, this piece of code exist?".

When links can be followed in both directions, traceability is called bidirectional. In a test management system a link between a requirement and a test case is usually one record, and it is read in both directions.

Gotel and Finkelstein also separated traceability by stage. Pre-requirements specification traceability links a requirement with its origins: the people, documents and decisions it comes from. Post-requirements specification traceability links a requirement with what follows from it: design, implementation, tests. Their study of more than 100 practitioners found that most problems attributed to poor traceability came from the first kind. Test management deals mostly with the second.

Chain of links

In testing, traceability forms a chain from the requirement to the defect:

Requirement PAY-7 "An expired card is rejected"
 |
 +-- Test condition: payment with an expired saved card
       |
       +-- Test case TC-13 "Pay with an expired card"
             |
             +-- Result in run "Release 2.4.0, Safari": failed
             |     |
             |     +-- Defect BUG-310
             |
             +-- Result in run "Retest, 2.4.1, Safari": passed

Each link in the chain is created by a different activity. The link between a requirement and a test condition appears during test analysis, the link between a test condition and a test case during test design, the link between a test case and a result during execution, the link between a result and a defect during defect reporting. The chain is complete only if every activity records its link.

The granularity of the links matters. A link between a whole specification and a test suite says nothing about coverage. The links are useful when they connect individual, identified items: a numbered requirement, a user story, an acceptance criterion, with individual test cases.

Traceability matrix

A traceability matrix is a table that shows the links between two kinds of items, for example requirements and test cases. Rows are items of one kind, columns of the other, a mark at the intersection means a link. The example below covers five requirements and six test cases:

Requirement	TC-11	TC-12	TC-13	TC-14	TC-15
PAY-5 Pay with a new card	x
PAY-6 Pay with a saved card		x
PAY-7 Reject an expired card			x	x
PAY-8 Refund a payment
PAY-9 Receipt by email		x			x

Two kinds of gaps can be read from the matrix directly:

a row without marks is a requirement without tests: PAY-8 is not covered;
a column without marks is a test case without a requirement: TC-16 is not linked to any of the listed requirements. It can check a requirement that is missing from the list, or it can be an obsolete test.

A test management system stores the links, not the matrix. The matrix is one of the views computed from the links, together with lists of uncovered requirements, orphan test cases and the status of each requirement in a given run.

Many-to-many links

Links between requirements and test cases are many-to-many. A requirement is usually checked by several test cases: PAY-7 has TC-13 for the rejection itself and TC-14 for the message shown to the user. A test case can check several requirements: TC-12 pays with a saved card and also checks that the receipt arrives by email, so it is linked to PAY-6 and PAY-9.

This affects how the status of a requirement in a run is derived from the results of its test cases. A common rule is:

the requirement failed if at least one of its linked test cases failed;
the requirement passed if all of its linked test cases were executed and passed;
otherwise the requirement is not fully tested: some of its test cases are blocked, skipped or not executed yet.

Under this rule a failure of TC-12 marks two requirements as failed at once. A report on requirements and a report on test cases therefore give different numbers for the same run, and both are correct.

Requirements coverage

Requirements coverage is the share of requirements that are covered by tests. In general terms, the ISTQB glossary defines coverage as the degree to which specified coverage items are exercised by a test suite, expressed as a percentage. For requirements coverage, the coverage items are the requirements.

The figure depends on what "covered" means, and there are three common variants:

Variant	A requirement is covered when	Question it answers
Linked	It has at least one linked test case	Is there a test for it at all?
Executed	At least one linked test case has been executed in the run	Has it been checked on this build?
Passed	All linked test cases executed in the run have passed	Does it work on this build?

For the matrix above and a run in which TC-11, TC-12, TC-13 and TC-15 passed and TC-14 failed:

linked coverage: 4 of 5 requirements have a test case (PAY-8 has none), 80 percent;
executed coverage: the same 4 requirements have an executed test case, 80 percent;
passed coverage: PAY-5, PAY-6 and PAY-9 have all their test cases passed, PAY-7 has a failed one, so 3 of 5, 60 percent.

A report states which variant it uses.

Limits of requirements coverage

Requirements coverage counts links, not the depth of checking. A requirement with one superficial test case and a requirement with twenty test cases covering all boundaries both count as covered. A coverage of 100 percent means that there is no requirement without tests. It does not mean that the requirements are tested enough.

All requirements are counted with equal weight. In practice coverage is often reported separately for requirements of different priority or risk: "all High-risk requirements are covered and passed" carries more information than "92 percent of requirements are covered".

The figure is only as complete as the list of requirements. Behavior that nobody wrote down as a requirement does not appear in it, however important it is. Techniques for deriving test cases that cover a requirement in depth are described in the series on test design, starting with Equivalence partitioning and boundary values.

Impact analysis

Impact analysis is, in the wording of the ISTQB glossary, the identification of all work products affected by a change, including an estimate of the resources needed to accomplish the change. In testing it answers the question: which tests have to be updated and executed again after this change?

With traceability, impact analysis follows the links:

Change: requirement PAY-7 now also rejects cards that expire this month
 |
 +-- Linked test cases: TC-13, TC-14       -> review and update
 +-- Their open defects: none
 +-- Related requirements: PAY-6           -> its test case TC-12 goes to regression

Without the links, the affected tests are found by searching the texts of test cases, which misses tests that describe the behavior in other words.

The ISTQB syllabus connects impact analysis with regression testing: impact analysis helps to determine the extent of regression testing after a change. Links between requirements and tests give one source of impact. Another source is the code: tools that record which tests execute which parts of the code can select the tests affected by a change in the code. The two sources complement each other: the first captures changes of intent, the second changes of implementation. Retesting and regression testing in test runs are described in the article Test run.

Keeping traceability up to date

Links lose their value when they are not maintained. The common causes are known in advance:

unstable identifiers: a requirement renumbered after a reorganization of the specification breaks all its links;
changed requirements: a link to a requirement whose text has changed still exists, although the test case may no longer match it;
links created late: links added before an audit, from memory, rather than during test design.

The practices that address them follow from the causes. Requirements and test cases have identifiers that do not change. A link is created by the person who designs the test case, at the moment of design. When a requirement changes, its links are marked for review until someone confirms that the test cases still match it. The requirement version a test case was written for is recorded.

Requirements often live in a different tool from test cases: a requirements management tool, a task tracker, a document. For the exchange of requirements between tools from different vendors the Object Management Group (OMG) publishes the Requirements Interchange Format (ReqIF), an XML-based format; its current version 1.2 was published in July 2016. Each requirement in a ReqIF file has an identifier that stays the same across exchanges, which makes it possible to keep links when requirements are re-imported.

Requirements coverage and code coverage

Requirements coverage and code coverage are both called coverage, but they measure different things.

	Requirements coverage	Code coverage
Coverage items	Requirements, user stories, acceptance criteria	Statements, branches, conditions of the code
How it is measured	From links between requirements and test cases, and from results	By instrumenting the code while tests run
What a gap means	A required behavior has no test	Some code is not executed by any test
What it cannot show	Code that implements no requirement; depth of checking	Missing functionality; whether the results are checked correctly
Typical level	System and acceptance tests	Unit and integration tests

Each measure finds gaps that the other cannot find. Code coverage cannot find a requirement that was never implemented, because there is no code to cover. Requirements coverage cannot find code that no test executes. The two are used together: requirements coverage for the question "is everything that is required checked?", code coverage for the question "is everything that is written executed by tests?". Test levels and their typical measures are described in the article Test automation levels and the test pyramid.

Terms

Term	Meaning
Requirements traceability	Ability to describe and follow the life of a requirement in both a forwards and backwards direction
Traceability	Degree to which a relationship can be established between two or more products of the development process
Forward traceability	Links from a requirement to the work products derived from it: design, code, tests, results
Backward traceability	Links from a work product back to the requirement it comes from
Traceability matrix	Table that shows the links between two kinds of items, for example requirements and test cases
Requirements coverage	Share of requirements covered by tests: linked, executed or passed
Coverage	Degree to which specified coverage items are exercised by a test suite, expressed as a percentage
Impact analysis	Identification of the work products affected by a change
ReqIF	Requirements Interchange Format: XML format of the OMG for exchanging requirements between tools