Pairwise testing

Pairwise testing is a black-box test technique in which test cases are designed to exercise pairs of parameter-value pairs: every value of every parameter appears at least once together with every value of every other parameter. It belongs to combinatorial testing, the group of techniques in which test cases exercise specific combinations of values of several parameters. Pairwise testing reduces the number of test cases from the product of all value counts to a number that grows slowly with the number of parameters.

The technique comes from statistical design of experiments. In 1985 Robert Mandl described in Communications of the ACM the use of orthogonal Latin squares to design tests for the Ada Compiler Validation Capability. According to NIST, orthogonal arrays have been used for pairwise black-box testing of software since the 1980s. In 2004 D. Richard Kuhn, Dolores Wallace and Albert Gallo published data on how many parameters take part in real software failures. Today the standard ISO/IEC/IEEE 29119-4:2021 describes combinatorial test design techniques in clause 5.2.5. The ISTQB Advanced Level Test Analyst syllabus (version 4.0, 2025, section 3.1.2) defines pairwise coverage; the Foundation Level syllabus 4.0.1 does not include the technique.

The problem pairwise testing solves

The number of combinations of parameter values is the product of the numbers of values. A web form tested on three browsers, three operating systems, three payment methods and two languages has 3 x 3 x 3 x 2 = 54 combinations. Each new parameter multiplies the count. NIST describes a panel of 34 on/off switches: exhaustive testing would need 2 to the power of 34 settings, about 1.7 x 10^10.

Testing all combinations is therefore impossible for most real configurations. Testing each value once, as Each Choice coverage does, is cheap but checks no interaction at all: a defect that appears only with Safari and invoice payment can be missed. Pairwise testing is a middle point. It assumes that most failures are caused by one parameter or by the interaction of two, and covers all such interactions with a small set of test cases.

The saving grows with the number of parameters. According to NIST SP 800-142, the number of tests needed to cover all t-way combinations grows in proportion to v^t log n, where v is the number of values per parameter and n is the number of parameters. The product of all values grows as v^n.

Parameters, values and pairs

The input of the technique is a list of parameters and their values. A parameter is an input or a configuration setting: browser, operating system, payment method, user role. Its values are usually the equivalence partitions of the parameter, not every possible value: "card" stands for all card types if they are processed the same way. How to choose them is described in equivalence partitioning and boundary values.

A parameter-value pair is a parameter together with one of its values, for example (Browser, Safari). In the Test Analyst syllabus, the coverage items of pairwise testing are pairs of parameter-value pairs for any two parameters, for example (Browser, Safari) with (Payment, Invoice). Coverage is the number of such pairs exercised by the tests divided by the number of all such pairs.

The number of coverage items is the sum over all pairs of parameters of the product of their value counts. For the form above it is:

Each test case covers one pair of values for each pair of parameters, so six coverage items at once. This is why far fewer than 54 test cases are needed.

A worked example

The parameters of the example are:

The full product has 54 combinations. A pairwise set needs at least 9 test cases: the 9 pairs of Browser and OS values can only be covered by 9 different rows. The following 9 test cases cover all 45 pairs:

The set can be checked pair by pair. Safari meets Invoice in PW-7, Linux meets de in PW-6, Wallet meets de in PW-2. Every value of Browser meets every value of OS and of Payment exactly once, and both languages appear with every value of the other parameters.

The table defines only the inputs. Each row becomes a test case with its own expected result, taken from the test basis: the order is placed, the confirmation page is in the selected language, the invoice is offered or not. Combinations known to be risky are added to the generated set by hand.

Orthogonal arrays and covering arrays

Two mathematical objects are used to build combinatorial test sets. An orthogonal array of strength t is a table in which every combination of values of any t columns appears the same number of times. It comes from design of experiments, where balance is needed to estimate the effect of each factor statistically. A covering array of strength t is a table in which every combination of values of any t columns appears at least once. Kuhn, Kacker and Lei describe covering-array test sets as unbalanced and note that statistical models are not used in testing based on them.

Four parameters with three values each have a well-known orthogonal array of strength 2 with 9 rows. The full product has 81 combinations, the array has 9 rows, and each of the 54 pairs of values appears in exactly one row. The table of the worked example is built from the same array, but Language has only two values. The fourth column of the array has three values, and its third value was replaced by en, so en appears in six rows and de in three. Every pair is still covered, but not equally often: the table is a covering array, not an orthogonal one.

Balance is not needed to find failures. A test either triggers the failure or not, and a pair tested twice gives no more coverage than a pair tested once. Pairwise test generation tools therefore build covering arrays: dropping the balance requirement leaves more freedom to reduce the number of rows.

How a pairwise set is built

The Test Analyst syllabus notes that finding a minimal set of test cases that achieves pairwise coverage is generally difficult. Small sets like the example can be built by hand from a known orthogonal array. Larger sets are generated by tools that use heuristics, so the size of the result can differ from tool to tool.

The work goes in steps:

• Parameters and values. The parameters are taken from the test basis, the values are partitions of each parameter.
• Constraints. Combinations that cannot occur are written down as rules.
• Generation. A tool builds the covering array of the required strength.
• Seeds. Combinations that must be tested, for example the most common configuration or one from a defect report, are added.
• Expected results. Each row receives its expected result from the test basis.

Microsoft PICT is a command-line generator published on GitHub. Its model file lists one parameter per line in the form Name: value1, value2, and constraints in the form IF [Parameter] = "value" THEN ...;. The default order of combinations is 2, set with the option /o. The model of the worked example with one constraint:

cat > checkout.txt <<'EOF'
Browser:  Chrome, Firefox, Safari
OS:       Windows, macOS, Linux
Payment:  Card, Wallet, Invoice
Language: en, de

IF [Browser] = "Safari" THEN [OS] = "macOS";
EOF

pict checkout.txt /o:2

PICT writes the test cases as tab-separated lines with a header row of parameter names. Seeding rows are given in a separate file. NIST publishes ACTS, a covering array generator that also supports constraints and strengths higher than 2; according to its project page, version 3.3 is the latest, and the tool is free and sent on request.

Constraints between parameters

Real parameters are rarely independent. Safari runs only on macOS; an invoice payment can be unavailable for guest users; a feature can exist only in one language. A constraint is a rule that excludes combinations that cannot occur or are not allowed. Without constraints the generator puts such combinations into test cases, and these test cases cannot be executed: PW-7 and PW-9 of the example run Safari on Windows and Linux.

A constraint changes both the full product and the pairs to be covered. With the rule "Safari only on macOS":

The pairs (Safari, Windows) and (Safari, Linux) disappear, but the minimum set grows. Safari has to meet three payment methods, and all these rows are now on macOS. Windows and Linux need three rows each for the three payment methods, with Chrome or Firefox. Chrome and Firefox also need a row with macOS each. That gives 3 + 3 + 3 + 2 = 11 rows:

A constraint is a statement about the system. If the shop can be opened in Safari on another platform, the rule is wrong, and the combination it removes is exactly the one that is not tested.

Invalid values are a separate case. A negative test checks a value outside the allowed range, such as an expired card. Two invalid values in one test case can mask each other: the first error stops the processing, and the second is never checked. PICT marks invalid values with the prefix ~ and puts at most one of them into a test case.

Higher strength and the limits of pairwise testing

Pairwise testing is t-way testing with t = 2. T-way testing covers every combination of values of any t parameters; t is called the strength. The empirical question is how large t must be. Kuhn, Wallace and Gallo analyzed failure reports of several kinds of software:

The authors conclude that pairwise testing is often effective but not always sufficient, and that a strength between 4 and 6 is needed for testing close to exhaustive, depending on dependability requirements. NIST summarizes its research in the same way: most failures are caused by one or two parameters, progressively fewer by three or more, and the observed failures were triggered by interactions of 1 to 6 parameters. Higher strength is affordable: for the 34 switches, NIST reports 33 tests for all 3-way combinations and 85 for all 4-way combinations.

Pairwise testing is not enough in several cases:

• High risk. Software whose failure is costly is tested with strength 3 or higher.
• Sequences. Combinatorial testing is less effective when a failure depends on the order of events. Such behavior is tested with state transition testing.
• Known business rules. When the outcome is defined for specific combinations of conditions, a decision table covers all of them.
• Values inside a parameter. Pairwise testing uses the values it is given. Boundary values have to be among them, or the tests miss defects at the boundaries.

Pairwise testing and related techniques

Combinatorial techniques differ in what combinations they require. The table orders them from weakest to strongest and adds decision tables for comparison.

Each Choice coverage is described in the article on equivalence partitioning; it checks values, not interactions. In base choice coverage, as the Test Analyst syllabus describes it, one value of each parameter is chosen as the base, for example the most common one. The first test case uses all base values, and each further test case changes one parameter to one of its other values. It answers the question "does each deviation from the normal case work?", but does not combine two deviations.

Decision tables and pairwise testing rest on opposite assumptions. A decision table assumes that the outcome depends on a known combination of conditions and covers all such combinations of one rule. Pairwise testing assumes that the parameters are largely independent and that unknown interactions are mostly between two of them. The two are combined: a rule with dependent conditions goes into a decision table, the independent configuration parameters around it go into a pairwise set.

Terms

Sources

• ISTQB Certified Tester Advanced Level Test Analyst Syllabus v4.0, ISTQB, 2025(opens in a new tab)
• ISTQB Certified Tester Foundation Level Syllabus v4.0.1, ISTQB, 2024(opens in a new tab)
• ISTQB Glossary(opens in a new tab)
• ISTQB Glossary mirror: pairwise testing, imbus(opens in a new tab)
• ISO/IEC/IEEE 29119-4:2021, Software and systems engineering, Software testing, Part 4: Test techniques(opens in a new tab)
• Orthogonal Latin squares: an application of experiment design to compiler testing, R. Mandl, Communications of the ACM, vol. 28, no. 10, 1985(opens in a new tab)
• Software Fault Interactions and Implications for Software Testing, D. R. Kuhn, D. R. Wallace, A. M. Gallo, IEEE Transactions on Software Engineering, vol. 30, no. 6, 2004(opens in a new tab)
• Practical Combinatorial Testing, NIST SP 800-142, D. R. Kuhn, R. N. Kacker, Y. Lei, October 2010(opens in a new tab)
• Advanced Combinatorial Test Methods for System Reliability, D. R. Kuhn, R. N. Kacker, Y. Lei, IEEE Reliability Society 2010 Annual Technical Report(opens in a new tab)
• Automated Combinatorial Testing for Software (ACTS), NIST(opens in a new tab)
• PICT documentation, Microsoft, GitHub(opens in a new tab)