A test run is a single execution of a selected set of test cases on a specific version of the product in a specific environment. A run records what was tested, where, when and by whom, and contains one result for every test case included in it. Runs are the unit in which a test management system records the execution of tests, both manual and automated.
The term comes from practice rather than from a standard. The ISTQB glossary uses it in the definition of a test suite: a set of test scripts or test procedures to be executed in a specific test run. The ISO/IEC/IEEE 29119-3 standard (test documentation, current edition 2021) does not define a test run document. It describes the records that a run produces: actual results and test results, the test execution log, test incident reports, and on top of them the test status report and the test completion report.
What a test run is for
A test case describes a check. It does not say whether the check was performed on the current build or what the outcome was. The same test case is executed many times: on every release candidate, on several browsers, after every fix. Each execution happens in a context: a version of the product, an environment, a date, an executor.
A test run fixes this context once for a whole set of executions. Its results answer the questions that a test case cannot answer:
- which test cases were executed on build 2.4.1 and which were not;
- how many of them passed, failed or could not be executed;
- whether the failures are new or were already present on the previous build;
- whether the exit criteria of the test level are met.
The basic concepts of a test management system (test case, suite, run, result) are introduced in the article What is a test management system.
Composition of a test run
Selection of test cases
A run includes the test cases that are needed for its goal. The usual ways to select them:
- a whole test suite or several suites, for example all tests of the checkout feature;
- a filter on attributes: all active test cases of priority "High" for the component "Payment";
- a regression subset, a fixed list of test cases executed on every release;
- the test cases affected by a change, selected through links to the changed requirements;
- the test cases that failed in a previous run, for retesting.
When a run is created, its list of test cases is fixed. Test cases added to the suite later do not appear in an existing run. This keeps the run comparable to its plan: the number of planned executions does not change during execution.
Version under test
A run refers to a specific version of the product: a build number, a release, a commit. Without it the results cannot be interpreted. A failed result on build 2.4.0 and a passed result on build 2.4.1 describe a fix. The same results without versions describe nothing.
Test environment
A test environment is, in the wording of the ISTQB glossary, an environment containing hardware, instrumentation, simulators, software tools and other support elements needed to conduct a test. For a run it is recorded as a set of parameters: operating system, browser and its version, device, server stand, configuration of the product, versions of external services.
When the same set of test cases has to be executed in several environments, two approaches are used. The first is a separate run for each environment. The second is one run with a configuration matrix, in which every test case gets one result per configuration:
Run "Release 2.4.1, regression"
|
+-- TC-12 "Pay with a saved card"
| +-- Windows 11, Chrome 141 -> passed
| +-- macOS 15, Safari 26 -> failed
| +-- Android 16, app -> passed
|
+-- TC-13 "Pay with an expired card"
+-- Windows 11, Chrome 141 -> passed
+-- macOS 15, Safari 26 -> passed
+-- Android 16, app -> blockedIn both cases a result belongs to exactly one environment. A result "passed" without an environment does not say where the behavior was checked.
Assignees and schedule
A run of manual tests is distributed between testers: each test case or group of test cases has an assignee. A run has a planned start and end, often tied to a milestone or a release date. Automated runs are started by a CI system and have no assignees, only a source: the pipeline and the job that produced the results.
Result statuses
The status of a result says what happened with one test case in one run. In ISO/IEC/IEEE 29119 the test result is reduced to two values: whether the test case passed or failed. Test management systems use a wider set, because not every planned test case reaches a verdict. Standards do not fix this set. A typical one:
| Status | Meaning |
|---|---|
| Untested | The test case is in the run but has not been executed yet |
| Passed | The actual result matches the expected result |
| Failed | The actual result differs from the expected result |
| Blocked | The test case cannot be executed because its preconditions are not met, for example because of another defect or an unavailable environment |
| Skipped | The test case was deliberately not executed in this run |
| Retest | The test case has to be executed again, for example after a fix |
"Failed" and "blocked" are different statuses for a reason. A failed test case has checked the behavior and found a deviation. A blocked test case has not checked anything. The ISTQB glossary describes a blocked test case as a test case that cannot be executed because the preconditions for its execution are not fulfilled. If blocked results are counted as failed, the report shows defects where there is a lack of information.
The results of automated tests come with their own statuses. In the JUnit XML format a test case without a result element is passed, failure means a failed check, error an unexpected problem during execution, skipped a test that was not executed. When results are imported, these statuses are mapped to the statuses of the system:
| JUnit XML | Status in the run |
|---|---|
| no result element | Passed |
failure |
Failed |
error |
Failed or blocked, depending on the rules of the team |
skipped |
Skipped |
Executing a run
During execution the tester performs the steps of each test case and records the result. A result contains more than the status:
- the actual result: what was observed, in particular when it differs from the expected result;
- attachments: screenshots, screen recordings, logs, files that show the observed behavior;
- the time spent and the executor;
- for a failed result, a link to a defect.
An anomaly, a difference between the actual and the expected result, is analyzed before a defect report is created. The ISTQB Foundation Level syllabus (version 4.0.1) notes that a reported anomaly can turn out to be a real defect, a false positive or a change request. A false positive is often caused by an error in the test case itself or by a problem in the environment. The structure of a defect report is described in the article Defect report.
A run is closed when all planned test cases have a final status or when the team decides to stop execution. After closing, the results of a run are not changed: they record the state of the product at a moment in time. A new execution is done in a new run.
Retesting and regression testing
After defects are fixed, two kinds of testing follow. The ISTQB syllabus separates them.
Confirmation testing, also called retesting, confirms that an original defect has been fixed. It executes again the test cases that failed because of the defect, on the build with the fix. Depending on the risk, new test cases can be added to check the fix.
Regression testing confirms that a change, including a fix, has not caused adverse consequences in other parts of the product. It executes test cases that passed before and that may be affected by the change. Its scope is determined with the help of impact analysis: the identification of the work products affected by a change. The links between requirements, test cases and code used for impact analysis are described in the article Requirements traceability and coverage.
In a test management system both kinds of testing are new runs on a new build:
Run A: "Release 2.4.0" TC-12 failed -> BUG-310
TC-40 passed
TC-41 passed
Fix of BUG-310 in build 2.4.1
Run B: "Retest BUG-310, 2.4.1" TC-12 passed (confirmation testing)
Run C: "Regression, 2.4.1" TC-40 passed (regression testing)
TC-41 failed -> BUG-325Regression test suites are executed many times and change slowly. They are the first candidates for automation. Automated regression runs in CI are described in the article Automated tests in CI.
Summary of a test run
The summary of a run is computed from its results. It shows the number of results in each status, the share of executed test cases and the share of passed ones. For a run with 200 planned test cases:
| Status | Number |
|---|---|
| Passed | 162 |
| Failed | 14 |
| Blocked | 9 |
| Skipped | 5 |
| Untested | 10 |
| Total | 200 |
From these numbers: 176 test cases reached a verdict, 162 passed and 14 failed. Blocked, skipped and untested test cases were not executed. 176 of 200 is 88 percent of the run. 162 of the 176 test cases with a verdict passed, which is 92 percent. The same run can be described as "81 percent passed" (162 of 200) or "92 percent passed" (162 of 176), depending on the denominator. A report states which denominator it uses. How metrics like these are computed and where they mislead is described in the article Test metrics.
The ISTQB syllabus distinguishes two kinds of reports based on results. A test progress report is produced regularly during testing: progress against the schedule, impediments, metrics, new risks and the testing planned for the next period. A test completion report is produced at a milestone: a summary of the testing performed, an evaluation against the exit criteria of the test plan, deviations from the plan, unfixed defects and residual risks, lessons learned. In a test management system both are assembled from the results of one or more runs.
The summary of a run is compared with the exit criteria of the test plan. A criterion such as "all test cases of priority High are executed, no open defects of severity Critical" is checked directly against the results and the linked defects. Exit criteria are described in the article Test plan and test strategy.
Test run, test suite and test plan
The three entities are related and are often confused.
| Test suite | Test run | Test plan | |
|---|---|---|---|
| What it is | Group of test cases | Execution of selected test cases | Description of the scope, approach and schedule of testing |
| Contains | Test cases | Results of test cases | Goals, criteria, references to runs |
| Changes over time | Test cases are added and removed | The list is fixed at creation, results are added during execution | Is revised when the scope or the schedule changes |
| Bound to a version of the product | No | Yes | To a release or a project |
| Lifetime | As long as the feature exists | Until it is closed; then it is kept as history | Until the end of the release or the project |
A suite answers the question "what can be tested", a run "what was tested on this build and with what outcome", a plan "what has to be tested, how and by when". A plan typically groups several runs: on different environments, for different test levels, or successive runs of one release.
Terms
| Term | Meaning |
|---|---|
| Test run | Execution of a selected set of test cases on a specific version of the product in a specific environment |
| Test result | Record of one execution of one test case within a test run |
| Test environment | Environment containing hardware, instrumentation, simulators, software tools and other support elements needed to conduct a test |
| Actual result | Behavior observed when a test case is executed |
| Blocked test case | Test case that cannot be executed because the preconditions for its execution are not fulfilled |
| Confirmation testing | Testing that confirms that an original defect has been fixed; also called retesting |
| Regression testing | Testing that confirms that a change has not caused adverse consequences in other parts of the product |
| Impact analysis | Identification of the work products affected by a change |
| Test progress report | Report produced regularly during testing: progress, impediments, metrics, risks |
| Test completion report | Report produced at a milestone: summary of testing, evaluation against exit criteria, residual risks |