Defect report

A defect report is a record of a defect found in a product: what was observed, under which conditions, how to reproduce it, how serious it is and in which state of handling it is. A defect report is written when the actual result of a test differs from the expected result and the analysis shows that the cause is in the product. Informally it is called a bug report.

The content of a defect report is described in the ISTQB Foundation Level syllabus (version 4.0.1, section 5.5, defect management). ISO/IEC/IEEE 29119-3, the international standard for test documentation (current edition 2021), describes the same record as a test incident report, and its predecessor IEEE 829 as an anomaly report. IEEE 1044-2009 defines a uniform approach to the classification of software anomalies; since March 2020 this standard has the status Inactive-Reserved.

What a defect report is for

The ISTQB syllabus names three objectives of defect reports:

• to give those responsible for handling and resolving the defect enough information to resolve the issue;
• to provide a means of tracking the quality of the work product;
• to provide ideas for improving the development and test process.

The first objective is the direct one. A developer who receives a defect report usually has not seen the failure. The report has to let them observe the same behavior and understand how it differs from what is expected, without additional questions. Every question that the report does not answer costs a round trip between the tester and the developer.

The two other objectives work across many reports. The number of open defects, their severities and the time they take to fix describe the quality of a release. The causes of defects and the phases in which they are introduced and found show where the process lets them through. These figures are described in the article Test metrics.

Error, defect and failure

The ISTQB glossary separates three concepts that are often mixed up:

• an error is a human action that produces an incorrect result, for example a misunderstanding of a requirement;
• a defect is an imperfection or deficiency in a work product where it does not meet its requirements or specifications, for example a wrong condition in the code;
• a failure is an event in which a component or system does not perform a required function within specified limits, for example an order that is created with a wrong amount.

An error can lead to a defect, and a defect, when the faulty code is executed under certain conditions, can lead to a failure. A tester observes failures. A defect report describes a failure and the conditions in which it occurs. The defect itself, the place in the code or in a document, is usually found by the developer during analysis.

Not every observed difference is a defect. The ISTQB syllabus notes that a reported anomaly can turn out to be a real defect, a false positive (for example, an error in the test case or a problem in the environment) or a change request (the behavior matches the requirement, but the requirement has to change). The defect management process resolves which of these it is.

Contents of a defect report

The ISTQB syllabus lists the typical content of a defect report for a failure found in dynamic testing:

Some of these fields are filled in by the tool: the identifier, the date, the author and the initial status.

Title

The title is the line that people read in lists, notifications and reports. It names the observed behavior and the conditions under which it occurs: "Payment with an expired saved card creates an order with status Paid". A title like "Payment does not work" says neither what happens nor when.

Steps to reproduce

Steps to reproduce are a sequence of actions that leads from a known starting state to the failure. They state the preconditions (the account, the data, the settings), each action and the place where the failure appears. The steps are minimal: actions that do not affect the result are removed. When the failure was found by a test case, the steps refer to it and add what differs.

Expected and actual results

The expected result refers to its source: a requirement, a specification, a design, the test case. The actual result describes what was observed, without interpretation: "The order list shows order 1042 with status Paid", not "The system ignores the card check".

Environment and attachments

The environment includes everything that can affect the behavior: the build of the product, the operating system, the browser and its version, the device, the server stand, relevant settings. Attachments show the failure: a screenshot or a recording for interface failures, a fragment of a log with the time and the request identifier for server failures, a file that causes the failure for import failures.

An example of a complete report:

Severity and priority

Severity and priority are two different attributes, and they are set by different people for different reasons.

Severity is the degree of impact of the defect on the interests of stakeholders or on the requirements. It describes the consequences of the failure: data loss, financial impact, impossibility to complete a task, a cosmetic difference. Severity is usually set by the tester and changes only when new information about the consequences appears.

Priority is the urgency of fixing the defect. It depends on severity but also on other factors: how many users are affected, whether a workaround exists, when the next release is, what the business plans are. Priority is usually set by whoever decides on the order of work: a product owner, a project manager, a triage meeting.

The scales of both attributes are not standardized and are set by each team. The combinations below show why they are kept separate:

Priority changes over time more often than severity. A deferred defect of low priority gets a high priority when a large customer reports it.

Defect life cycle

A defect report moves through states from creation to closing. The ISTQB syllabus describes the process in four steps: the anomaly is logged; it is analyzed and classified; a response is decided, for example to fix it or keep it as it is; the report is closed. The syllabus gives examples of statuses: open, deferred, duplicate, waiting to be fixed, awaiting confirmation testing, re-opened, closed, rejected.

A common sequence of states:

           +--> Rejected (not a defect: false positive, works as specified)
           +--> Duplicate (already reported)
           +--> Deferred (will not be fixed now)
           |
Open --> Analyzed --> Waiting to be fixed --> Awaiting confirmation testing --> Closed
                                ^                         |
                                |                         v
                                +-------------------- Re-opened

Two transitions are worth noting. A report goes to "awaiting confirmation testing" after the fix, not to "closed": the fix is confirmed by executing the test again on the build that contains it. If confirmation testing fails, the report is re-opened. These steps are described in the article Test run.

A rejected or duplicate report is not deleted. It records that the behavior was reported and why it is not handled as a defect, and the next person who observes the same behavior finds it.

The states and transitions are set by the team and by the tool. A bug tracker usually enforces them as a workflow: who can move a report from which state to which.

Reproducibility

A defect that reproduces on every attempt is the easiest to fix: the developer observes it, finds the cause and checks the fix. A defect that reproduces only sometimes needs more information in the report:

• the share of attempts in which the failure occurred, for example "3 of 10";
• the conditions that were the same and those that varied: data, time of day, load, order of actions;
• logs with timestamps for both successful and failed attempts;
• whether the failure reproduces in another environment.

A report of a failure that was observed once and could not be reproduced is still written. It states that the failure could not be reproduced and contains everything that was recorded at the moment: the time, the logs, the screenshot. Several such reports about the same area can point to one cause.

Intermittent failures of automated tests are a separate subject: such a test can be unreliable itself rather than detect a defect. Flaky tests are described in the article Flaky tests.

Links between a defect, a test case and a test run

A defect found during testing is linked in both directions:

Requirement PAY-7
 |
 +-- Test case TC-13 "Pay with an expired card"
       |
       +-- Result in run "Release 2.4.0, Safari": failed --> BUG-310
       +-- Result in run "Retest BUG-310, 2.4.1": passed --> BUG-310 closed

The links serve several purposes:

• from a failed result, the defect shows what has been done about the failure;
• from a defect, the result shows on which build and in which environment the failure was observed;
• from a defect, the test case gives the steps for confirmation testing;
• from a test case, the list of linked defects shows how often the area breaks;
• from a requirement, the open defects of its test cases show the state of the requirement for a release.

A defect found outside a test case, for example in exploratory testing or reported by a user, is a reason to add a test case. Once the defect is fixed, the test case is included in regression testing so that the same defect is detected if it returns.

Defect report and related records

A defect report is often kept in the same tool as other kinds of records, and the boundary between them affects counting and reports.

A change request reported as a defect inflates defect counts and makes the product look less stable than it is. A defect reported as a change request disappears from defect metrics. A production incident can be caused by a defect, and then a defect report is linked to the incident. It can also have another cause, such as a hardware failure or a configuration error.

Terms

Sources

• ISTQB Certified Tester Foundation Level Syllabus v4.0.1, ISTQB, 2024(opens in a new tab)
• ISTQB Glossary(opens in a new tab)
• ISO/IEC/IEEE 29119-3:2021, Software testing, Part 3: Test documentation(opens in a new tab)
• IEEE 1044-2009, Standard Classification for Software Anomalies(opens in a new tab)