[{"data":1,"prerenderedAt":42},["ShallowReactive",2],{"wiki-test-design-pairwise-testing-en":3},{"meta":4,"html":8,"headings":9,"words":41},{"title":5,"description":6,"published":7,"updated":7},"Pairwise testing","Pairwise testing: the combinatorial explosion, covering every pair of parameter values, covering and orthogonal arrays, constraints and t-way testing.","2026-11-03","\u003Cp>\u003Cstrong>Pairwise testing\u003C/strong> is a black-box test technique in which \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-case\">test cases\u003C/span> are designed to exercise pairs of parameter-value pairs: every value of every parameter appears at least once together with every value of every other parameter. It belongs to \u003Cstrong>combinatorial testing\u003C/strong>, the group of techniques in which test cases exercise specific combinations of values of several parameters. Pairwise testing reduces the number of test cases from the product of all value counts to a number that grows slowly with the number of parameters.\u003C/p>\n\u003Cp>The technique comes from statistical design of experiments. In 1985 Robert Mandl described in Communications of the ACM the use of orthogonal Latin squares to design tests for the Ada Compiler Validation Capability. According to NIST, orthogonal arrays have been used for pairwise black-box testing of software since the 1980s. In 2004 D. Richard Kuhn, Dolores Wallace and Albert Gallo published data on how many parameters take part in real software failures. Today the standard ISO/IEC/IEEE 29119-4:2021 describes combinatorial test design techniques in clause 5.2.5. The ISTQB Advanced Level Test Analyst syllabus (version 4.0, 2025, section 3.1.2) defines pairwise coverage; the Foundation Level syllabus 4.0.1 does not include the technique.\u003C/p>\n\u003Ch2 id=\"the-problem-pairwise-testing-solves\" class=\"wiki-heading\">The problem pairwise testing solves\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"the-problem-pairwise-testing-solves\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The number of combinations of parameter values is the product of the numbers of values. A web form tested on three browsers, three operating systems, three payment methods and two languages has 3 x 3 x 3 x 2 = 54 combinations. Each new parameter multiplies the count. NIST describes a panel of 34 on/off switches: exhaustive testing would need 2 to the power of 34 settings, about 1.7 x 10^10.\u003C/p>\n\u003Cp>Testing all combinations is therefore impossible for most real configurations. Testing each value once, as \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"each-choice-coverage\">Each Choice coverage\u003C/span> does, is cheap but checks no interaction at all: a \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"defect\">defect\u003C/span> that appears only with Safari and invoice payment can be missed. Pairwise testing is a middle point. It assumes that most failures are caused by one parameter or by the interaction of two, and covers all such interactions with a small set of test cases.\u003C/p>\n\u003Cp>The saving grows with the number of parameters. According to NIST SP 800-142, the number of tests needed to cover all t-way combinations grows in proportion to v^t log n, where v is the number of values per parameter and n is the number of parameters. The product of all values grows as v^n.\u003C/p>\n\u003Ch2 id=\"parameters-values-and-pairs\" class=\"wiki-heading\">Parameters, values and pairs\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"parameters-values-and-pairs\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The input of the technique is a list of parameters and their values. A parameter is an input or a configuration setting: browser, operating system, payment method, user role. Its values are usually the \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"equivalence-partition\">equivalence partitions\u003C/span> of the parameter, not every possible value: "card" stands for all card types if they are processed the same way. How to choose them is described in \u003Ca href=\"/en/wiki/test-design/equivalence-partitioning-and-boundary-values\" class=\"wiki-link wiki-link--article\">equivalence partitioning and boundary values\u003C/a>.\u003C/p>\n\u003Cp>A \u003Cstrong>parameter-value pair\u003C/strong> is a parameter together with one of its values, for example (Browser, Safari). In the Test Analyst syllabus, the coverage items of pairwise testing are pairs of parameter-value pairs for any two parameters, for example (Browser, Safari) with (Payment, Invoice). Coverage is the number of such pairs exercised by the tests divided by the number of all such pairs.\u003C/p>\n\u003Cp>The number of coverage items is the sum over all pairs of parameters of the product of their value counts. For the form above it is:\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-2\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Pair of parameters\u003C/th>\n\u003Cth>Pairs of values\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Browser and OS\u003C/td>\n\u003Ctd>3 x 3 = 9\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Browser and Payment\u003C/td>\n\u003Ctd>3 x 3 = 9\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Browser and Language\u003C/td>\n\u003Ctd>3 x 2 = 6\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>OS and Payment\u003C/td>\n\u003Ctd>3 x 3 = 9\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>OS and Language\u003C/td>\n\u003Ctd>3 x 2 = 6\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Payment and Language\u003C/td>\n\u003Ctd>3 x 2 = 6\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Total\u003C/td>\n\u003Ctd>45\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>Each test case covers one pair of values for each pair of parameters, so six coverage items at once. This is why far fewer than 54 test cases are needed.\u003C/p>\n\u003Ch2 id=\"a-worked-example\" class=\"wiki-heading\">A worked example\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"a-worked-example\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The parameters of the example are:\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-2\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Parameter\u003C/th>\n\u003Cth>Values\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Browser\u003C/td>\n\u003Ctd>Chrome, Firefox, Safari\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>OS\u003C/td>\n\u003Ctd>Windows, macOS, Linux\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Payment\u003C/td>\n\u003Ctd>Card, Wallet, Invoice\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Language\u003C/td>\n\u003Ctd>en, de\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>The full product has 54 combinations. A pairwise set needs at least 9 test cases: the 9 pairs of Browser and OS values can only be covered by 9 different rows. The following 9 test cases cover all 45 pairs:\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-5\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Test case\u003C/th>\n\u003Cth>Browser\u003C/th>\n\u003Cth>OS\u003C/th>\n\u003Cth>Payment\u003C/th>\n\u003Cth>Language\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>PW-1\u003C/td>\n\u003Ctd>Chrome\u003C/td>\n\u003Ctd>Windows\u003C/td>\n\u003Ctd>Card\u003C/td>\n\u003Ctd>en\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>PW-2\u003C/td>\n\u003Ctd>Chrome\u003C/td>\n\u003Ctd>macOS\u003C/td>\n\u003Ctd>Wallet\u003C/td>\n\u003Ctd>de\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>PW-3\u003C/td>\n\u003Ctd>Chrome\u003C/td>\n\u003Ctd>Linux\u003C/td>\n\u003Ctd>Invoice\u003C/td>\n\u003Ctd>en\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>PW-4\u003C/td>\n\u003Ctd>Firefox\u003C/td>\n\u003Ctd>Windows\u003C/td>\n\u003Ctd>Wallet\u003C/td>\n\u003Ctd>en\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>PW-5\u003C/td>\n\u003Ctd>Firefox\u003C/td>\n\u003Ctd>macOS\u003C/td>\n\u003Ctd>Invoice\u003C/td>\n\u003Ctd>en\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>PW-6\u003C/td>\n\u003Ctd>Firefox\u003C/td>\n\u003Ctd>Linux\u003C/td>\n\u003Ctd>Card\u003C/td>\n\u003Ctd>de\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>PW-7\u003C/td>\n\u003Ctd>Safari\u003C/td>\n\u003Ctd>Windows\u003C/td>\n\u003Ctd>Invoice\u003C/td>\n\u003Ctd>de\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>PW-8\u003C/td>\n\u003Ctd>Safari\u003C/td>\n\u003Ctd>macOS\u003C/td>\n\u003Ctd>Card\u003C/td>\n\u003Ctd>en\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>PW-9\u003C/td>\n\u003Ctd>Safari\u003C/td>\n\u003Ctd>Linux\u003C/td>\n\u003Ctd>Wallet\u003C/td>\n\u003Ctd>en\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>The set can be checked pair by pair. Safari meets Invoice in PW-7, Linux meets de in PW-6, Wallet meets de in PW-2. Every value of Browser meets every value of OS and of Payment exactly once, and both languages appear with every value of the other parameters.\u003C/p>\n\u003Cp>The table defines only the inputs. Each row becomes a test case with its own \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"expected-result\">expected result\u003C/span>, taken from the test basis: the order is placed, the confirmation page is in the selected language, the invoice is offered or not. Combinations known to be risky are added to the generated set by hand.\u003C/p>\n\u003Ch2 id=\"orthogonal-arrays-and-covering-arrays\" class=\"wiki-heading\">Orthogonal arrays and covering arrays\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"orthogonal-arrays-and-covering-arrays\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>Two mathematical objects are used to build combinatorial test sets. An \u003Cstrong>orthogonal array\u003C/strong> of strength t is a table in which every combination of values of any t columns appears the same number of times. It comes from design of experiments, where balance is needed to estimate the effect of each factor statistically. A \u003Cstrong>covering array\u003C/strong> of strength t is a table in which every combination of values of any t columns appears at least once. Kuhn, Kacker and Lei describe covering-array test sets as unbalanced and note that statistical models are not used in testing based on them.\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-3\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Property\u003C/th>\n\u003Cth>Orthogonal array\u003C/th>\n\u003Cth>Covering array\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Each t-way combination appears\u003C/td>\n\u003Ctd>The same number of times\u003C/td>\n\u003Ctd>At least once\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Balance\u003C/td>\n\u003Ctd>Required\u003C/td>\n\u003Ctd>Not required\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Origin\u003C/td>\n\u003Ctd>Design of experiments\u003C/td>\n\u003Ctd>Combinatorial design, software testing\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Statistical model of the results\u003C/td>\n\u003Ctd>Used\u003C/td>\n\u003Ctd>Not used\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>Four parameters with three values each have a well-known orthogonal array of strength 2 with 9 rows. The full product has 81 combinations, the array has 9 rows, and each of the 54 pairs of values appears in exactly one row. The table of the worked example is built from the same array, but Language has only two values. The fourth column of the array has three values, and its third value was replaced by en, so en appears in six rows and de in three. Every pair is still covered, but not equally often: the table is a covering array, not an orthogonal one.\u003C/p>\n\u003Cp>Balance is not needed to find failures. A test either triggers the failure or not, and a pair tested twice gives no more coverage than a pair tested once. Pairwise test generation tools therefore build covering arrays: dropping the balance requirement leaves more freedom to reduce the number of rows.\u003C/p>\n\u003Ch2 id=\"how-a-pairwise-set-is-built\" class=\"wiki-heading\">How a pairwise set is built\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"how-a-pairwise-set-is-built\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The Test Analyst syllabus notes that finding a minimal set of test cases that achieves pairwise coverage is generally difficult. Small sets like the example can be built by hand from a known orthogonal array. Larger sets are generated by tools that use heuristics, so the size of the result can differ from tool to tool.\u003C/p>\n\u003Cp>The work goes in steps:\u003C/p>\n\u003Cul>\n\u003Cli>\u003Cstrong>Parameters and values.\u003C/strong> The parameters are taken from the test basis, the values are partitions of each parameter.\u003C/li>\n\u003Cli>\u003Cstrong>Constraints.\u003C/strong> Combinations that cannot occur are written down as rules.\u003C/li>\n\u003Cli>\u003Cstrong>Generation.\u003C/strong> A tool builds the covering array of the required strength.\u003C/li>\n\u003Cli>\u003Cstrong>Seeds.\u003C/strong> Combinations that must be tested, for example the most common configuration or one from a \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"defect-report\">defect report\u003C/span>, are added.\u003C/li>\n\u003Cli>\u003Cstrong>Expected results.\u003C/strong> Each row receives its expected result from the test basis.\u003C/li>\n\u003C/ul>\n\u003Cp>Microsoft PICT is a command-line generator published on GitHub. Its model file lists one parameter per line in the form \u003Ccode>Name: value1, value2\u003C/code>, and constraints in the form \u003Ccode>IF [Parameter] = "value" THEN ...;\u003C/code>. The default order of combinations is 2, set with the option \u003Ccode>/o\u003C/code>. The model of the worked example with one constraint:\u003C/p>\n\u003Cdiv class=\"wiki-code\" data-language=\"bash\">\u003Cdiv class=\"wiki-code__header\">\u003Cspan class=\"wiki-code__language\">Bash\u003C/span>\u003Cbutton type=\"button\" class=\"wiki-code__copy\" data-wiki-copy data-copied-label=\"Copied\">\u003Csvg class=\"wiki-code__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M216,28H88A12,12,0,0,0,76,40V76H40A12,12,0,0,0,28,88V216a12,12,0,0,0,12,12H168a12,12,0,0,0,12-12V180h36a12,12,0,0,0,12-12V40A12,12,0,0,0,216,28ZM156,204H52V100H156Zm48-48H180V88a12,12,0,0,0-12-12H100V52H204Z\"/>\u003C/svg>\u003Csvg class=\"wiki-code__icon wiki-code__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003Cspan class=\"wiki-code__label\">Copy\u003C/span>\u003Cspan class=\"wiki-code__label wiki-code__label--done\">Copied\u003C/span>\u003C/button>\u003C/div>\u003Cpre>\u003Ccode>cat > checkout.txt <<'EOF'\nBrowser:  Chrome, Firefox, Safari\nOS:       Windows, macOS, Linux\nPayment:  Card, Wallet, Invoice\nLanguage: en, de\n\nIF [Browser] = "Safari" THEN [OS] = "macOS";\nEOF\n\npict checkout.txt /o:2\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>PICT writes the test cases as tab-separated lines with a header row of parameter names. Seeding rows are given in a separate file. NIST publishes ACTS, a covering array generator that also supports constraints and strengths higher than 2; according to its project page, version 3.3 is the latest, and the tool is free and sent on request.\u003C/p>\n\u003Ch2 id=\"constraints-between-parameters\" class=\"wiki-heading\">Constraints between parameters\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"constraints-between-parameters\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>Real parameters are rarely independent. Safari runs only on macOS; an invoice payment can be unavailable for guest users; a feature can exist only in one language. A \u003Cstrong>constraint\u003C/strong> is a rule that excludes combinations that cannot occur or are not allowed. Without constraints the generator puts such combinations into test cases, and these test cases cannot be executed: PW-7 and PW-9 of the example run Safari on Windows and Linux.\u003C/p>\n\u003Cp>A constraint changes both the full product and the pairs to be covered. With the rule "Safari only on macOS":\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-3\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Count\u003C/th>\n\u003Cth>Without constraint\u003C/th>\n\u003Cth>With constraint\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Combinations in the full product\u003C/td>\n\u003Ctd>54\u003C/td>\n\u003Ctd>42\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Pairs of values to cover\u003C/td>\n\u003Ctd>45\u003C/td>\n\u003Ctd>43\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Minimum pairwise test cases\u003C/td>\n\u003Ctd>9\u003C/td>\n\u003Ctd>11\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>The pairs (Safari, Windows) and (Safari, Linux) disappear, but the minimum set grows. Safari has to meet three payment methods, and all these rows are now on macOS. Windows and Linux need three rows each for the three payment methods, with Chrome or Firefox. Chrome and Firefox also need a row with macOS each. That gives 3 + 3 + 3 + 2 = 11 rows:\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-5\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Test case\u003C/th>\n\u003Cth>Browser\u003C/th>\n\u003Cth>OS\u003C/th>\n\u003Cth>Payment\u003C/th>\n\u003Cth>Language\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>PC-1\u003C/td>\n\u003Ctd>Chrome\u003C/td>\n\u003Ctd>Windows\u003C/td>\n\u003Ctd>Card\u003C/td>\n\u003Ctd>en\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>PC-2\u003C/td>\n\u003Ctd>Firefox\u003C/td>\n\u003Ctd>Windows\u003C/td>\n\u003Ctd>Wallet\u003C/td>\n\u003Ctd>de\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>PC-3\u003C/td>\n\u003Ctd>Chrome\u003C/td>\n\u003Ctd>Windows\u003C/td>\n\u003Ctd>Invoice\u003C/td>\n\u003Ctd>de\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>PC-4\u003C/td>\n\u003Ctd>Firefox\u003C/td>\n\u003Ctd>Linux\u003C/td>\n\u003Ctd>Card\u003C/td>\n\u003Ctd>de\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>PC-5\u003C/td>\n\u003Ctd>Chrome\u003C/td>\n\u003Ctd>Linux\u003C/td>\n\u003Ctd>Wallet\u003C/td>\n\u003Ctd>en\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>PC-6\u003C/td>\n\u003Ctd>Firefox\u003C/td>\n\u003Ctd>Linux\u003C/td>\n\u003Ctd>Invoice\u003C/td>\n\u003Ctd>en\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>PC-7\u003C/td>\n\u003Ctd>Safari\u003C/td>\n\u003Ctd>macOS\u003C/td>\n\u003Ctd>Card\u003C/td>\n\u003Ctd>en\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>PC-8\u003C/td>\n\u003Ctd>Safari\u003C/td>\n\u003Ctd>macOS\u003C/td>\n\u003Ctd>Wallet\u003C/td>\n\u003Ctd>de\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>PC-9\u003C/td>\n\u003Ctd>Safari\u003C/td>\n\u003Ctd>macOS\u003C/td>\n\u003Ctd>Invoice\u003C/td>\n\u003Ctd>en\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>PC-10\u003C/td>\n\u003Ctd>Chrome\u003C/td>\n\u003Ctd>macOS\u003C/td>\n\u003Ctd>Card\u003C/td>\n\u003Ctd>de\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>PC-11\u003C/td>\n\u003Ctd>Firefox\u003C/td>\n\u003Ctd>macOS\u003C/td>\n\u003Ctd>Wallet\u003C/td>\n\u003Ctd>en\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>A constraint is a statement about the system. If the shop can be opened in Safari on another platform, the rule is wrong, and the combination it removes is exactly the one that is not tested.\u003C/p>\n\u003Cp>Invalid values are a separate case. A negative test checks a value outside the allowed range, such as an expired card. Two invalid values in one test case can mask each other: the first error stops the processing, and the second is never checked. PICT marks invalid values with the prefix \u003Ccode>~\u003C/code> and puts at most one of them into a test case.\u003C/p>\n\u003Ch2 id=\"higher-strength-and-the-limits-of-pairwise-testing\" class=\"wiki-heading\">Higher strength and the limits of pairwise testing\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"higher-strength-and-the-limits-of-pairwise-testing\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>Pairwise testing is t-way testing with t = 2. \u003Cstrong>T-way testing\u003C/strong> covers every combination of values of any t parameters; t is called the strength. The empirical question is how large t must be. Kuhn, Wallace and Gallo analyzed failure reports of several kinds of software:\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-2\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Data\u003C/th>\n\u003Cth>Finding\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Medical device software, FDA recalls\u003C/td>\n\u003Ctd>97 percent of the analyzed flaws could be detected by testing all pairs; the most complex needed four conditions\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Mozilla browser and Apache server\u003C/td>\n\u003Ctd>More than 70 percent of failures were triggered by one or two conditions; none needed more than six\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>NASA distributed database, after NIST SP 800-142\u003C/td>\n\u003Ctd>67 percent by one parameter, 93 percent by two, 98 percent by three\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>NASA Remote Agent planner\u003C/td>\n\u003Ctd>Pairs detected over 80 percent of correctness and convergence faults, about half of engine and interface faults\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>The authors conclude that pairwise testing is often effective but not always sufficient, and that a strength between 4 and 6 is needed for testing close to exhaustive, depending on dependability requirements. NIST summarizes its research in the same way: most failures are caused by one or two parameters, progressively fewer by three or more, and the observed failures were triggered by interactions of 1 to 6 parameters. Higher strength is affordable: for the 34 switches, NIST reports 33 tests for all 3-way combinations and 85 for all 4-way combinations.\u003C/p>\n\u003Cp>Pairwise testing is not enough in several cases:\u003C/p>\n\u003Cul>\n\u003Cli>\u003Cstrong>High risk.\u003C/strong> Software whose failure is costly is tested with strength 3 or higher.\u003C/li>\n\u003Cli>\u003Cstrong>Sequences.\u003C/strong> Combinatorial testing is less effective when a failure depends on the order of events. Such behavior is tested with \u003Ca href=\"/en/wiki/test-design/state-transition-testing\" class=\"wiki-link wiki-link--article\">state transition testing\u003C/a>.\u003C/li>\n\u003Cli>\u003Cstrong>Known business rules.\u003C/strong> When the outcome is defined for specific combinations of conditions, a \u003Ca href=\"/en/wiki/test-design/decision-table-testing\" class=\"wiki-link wiki-link--article\">decision table\u003C/a> covers all of them.\u003C/li>\n\u003Cli>\u003Cstrong>Values inside a parameter.\u003C/strong> Pairwise testing uses the values it is given. \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"boundary-value\">Boundary values\u003C/span> have to be among them, or the tests miss defects at the boundaries.\u003C/li>\n\u003C/ul>\n\u003Ch2 id=\"pairwise-testing-and-related-techniques\" class=\"wiki-heading\">Pairwise testing and related techniques\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"pairwise-testing-and-related-techniques\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>Combinatorial techniques differ in what combinations they require. The table orders them from weakest to strongest and adds \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"decision-table\">decision tables\u003C/span> for comparison.\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-3\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Technique\u003C/th>\n\u003Cth>What is covered\u003C/th>\n\u003Cth>Tests for the example\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Each Choice coverage\u003C/td>\n\u003Ctd>Every value of every parameter once\u003C/td>\n\u003Ctd>3\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Base choice coverage\u003C/td>\n\u003Ctd>A base combination and every single change of it\u003C/td>\n\u003Ctd>1 + 2 + 2 + 2 + 1 = 8\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Pairwise testing\u003C/td>\n\u003Ctd>Every pair of values of any two parameters\u003C/td>\n\u003Ctd>9\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>T-way testing, t = 3\u003C/td>\n\u003Ctd>Every triple of values of any three parameters\u003C/td>\n\u003Ctd>at least 27\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>All combinations\u003C/td>\n\u003Ctd>The full product\u003C/td>\n\u003Ctd>54\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Decision table testing\u003C/td>\n\u003Ctd>Every feasible combination of conditions of one rule\u003C/td>\n\u003Ctd>depends on the rule\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>Each Choice coverage is described in the article on \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"equivalence-partitioning\">equivalence partitioning\u003C/span>; it checks values, not interactions. In base choice coverage, as the Test Analyst syllabus describes it, one value of each parameter is chosen as the base, for example the most common one. The first test case uses all base values, and each further test case changes one parameter to one of its other values. It answers the question "does each deviation from the normal case work?", but does not combine two deviations.\u003C/p>\n\u003Cp>Decision tables and pairwise testing rest on opposite assumptions. A decision table assumes that the outcome depends on a known combination of conditions and covers all such combinations of one rule. Pairwise testing assumes that the parameters are largely independent and that unknown interactions are mostly between two of them. The two are combined: a rule with dependent conditions goes into a decision table, the independent configuration parameters around it go into a pairwise set.\u003C/p>\n\u003Ch2 id=\"terms\" class=\"wiki-heading\">Terms\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"terms\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cdiv class=\"wiki-table wiki-table--cols-2\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Term\u003C/th>\n\u003Cth>Meaning\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Pairwise testing\u003C/td>\n\u003Ctd>Black-box test technique in which test cases are designed to exercise pairs of parameter-value pairs\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Combinatorial testing\u003C/td>\n\u003Ctd>Black-box test technique in which test cases exercise specific combinations of values of several parameters\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Parameter-value pair\u003C/td>\n\u003Ctd>A parameter together with one of its values, for example (Browser, Safari)\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Orthogonal array\u003C/td>\n\u003Ctd>Table in which every combination of values of any t columns appears the same number of times\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Covering array\u003C/td>\n\u003Ctd>Table in which every combination of values of any t columns appears at least once\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Constraint\u003C/td>\n\u003Ctd>Rule that excludes combinations of parameter values that cannot occur or are not allowed\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>T-way testing\u003C/td>\n\u003Ctd>Combinatorial testing that covers every combination of values of any t parameters; t is the strength\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Base choice coverage\u003C/td>\n\u003Ctd>Coverage of a base combination of values and of every change of one parameter from it\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Ch2 id=\"sources\" class=\"wiki-heading\">Sources\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"sources\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cul>\n\u003Cli>\u003Ca href=\"https://www.istqb.org/wp-content/uploads/sdm-uploads/ISTQB-CTAL-TA-Syllabus-v4.0-EN-4.pdf\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">ISTQB Certified Tester Advanced Level Test Analyst Syllabus v4.0, ISTQB, 2025\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://www.istqb.org/wp-content/uploads/2024/11/ISTQB_CTFL_Syllabus_v4.0.1.pdf\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">ISTQB Certified Tester Foundation Level Syllabus v4.0.1, ISTQB, 2024\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://glossary.istqb.org/\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">ISTQB Glossary\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://www.imbus.de/en/glossar/term/pairwise-testing\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">ISTQB Glossary mirror: pairwise testing, imbus\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://www.iso.org/standard/79430.html\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">ISO/IEC/IEEE 29119-4:2021, Software and systems engineering, Software testing, Part 4: Test techniques\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://cacm.acm.org/research/orthogonal-latin-squares-an-application-of-experiment-design-to-compiler-testing/\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Orthogonal Latin squares: an application of experiment design to compiler testing, R. Mandl, Communications of the ACM, vol. 28, no. 10, 1985\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://webarchive.library.unt.edu/eot2008/20080916140037/http://csrc.nist.gov/staff/Kuhn/kuhn-wallace-gallo-04.pdf\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Software Fault Interactions and Implications for Software Testing, D. R. Kuhn, D. R. Wallace, A. M. Gallo, IEEE Transactions on Software Engineering, vol. 30, no. 6, 2004\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-142.pdf\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Practical Combinatorial Testing, NIST SP 800-142, D. R. Kuhn, R. N. Kacker, Y. Lei, October 2010\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://webarchive.library.unt.edu/web/20130217091851/http:/csrc.nist.gov/groups/SNS/acts/documents/ieee-rs-atr.pdf\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Advanced Combinatorial Test Methods for System Reliability, D. R. Kuhn, R. N. Kacker, Y. Lei, IEEE Reliability Society 2010 Annual Technical Report\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://csrc.nist.gov/projects/automated-combinatorial-testing-for-software\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Automated Combinatorial Testing for Software (ACTS), NIST\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://github.com/microsoft/pict/blob/main/doc/pict.md\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">PICT documentation, Microsoft, GitHub\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003C/ul>\n",[10,14,17,20,23,26,29,32,35,38],{"id":11,"level":12,"text":13},"the-problem-pairwise-testing-solves",2,"The problem pairwise testing solves",{"id":15,"level":12,"text":16},"parameters-values-and-pairs","Parameters, values and pairs",{"id":18,"level":12,"text":19},"a-worked-example","A worked example",{"id":21,"level":12,"text":22},"orthogonal-arrays-and-covering-arrays","Orthogonal arrays and covering arrays",{"id":24,"level":12,"text":25},"how-a-pairwise-set-is-built","How a pairwise set is built",{"id":27,"level":12,"text":28},"constraints-between-parameters","Constraints between parameters",{"id":30,"level":12,"text":31},"higher-strength-and-the-limits-of-pairwise-testing","Higher strength and the limits of pairwise testing",{"id":33,"level":12,"text":34},"pairwise-testing-and-related-techniques","Pairwise testing and related techniques",{"id":36,"level":12,"text":37},"terms","Terms",{"id":39,"level":12,"text":40},"sources","Sources",2610,1793718939787]