Talk:All-pairs testing

Replacing All-Pairs with Random Values ?

In one of the courses I have attended, someone argued that instead of running the all-pair combinations, one can run same number of combinations, with random values out of the vectors, and gain ~90% coverage in relation to all-pair coverage.

Should one really define pairs for testing if that is true ? It might be easier to execute O(nm) random cases, or if we would like to improve coverage, then O(nm) x 2.

Should this be indicated in the all-pair term ?

---

The suggestion that there is little benefit from pairwise test cases compared with randomly generated cases seems to come from one example in Bach and Schroeder's, “Pairwise Testing: A Best Practice that Isn’t.” https://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf See also the notes for slides #35-36 from Sherwood's review "Pairwise Testing Comes of Age." https://testcover.com/pub/background/stareast2008.ppt "In this example the pairwise test cases cover 100% of the required pairs. The authors observe that an equal number of random tests cover 88% of the pairs; therefore the pairwise tests aren’t much better. But in the same example it takes more than twice as many random test cases to cover all the pairs. If we’re trying to generate a small set of covering test cases, there is no real choice here." "In all the comparisons I’ve seen between pairwise and random generators, the pairwise tools are more efficient. That is, they produce smaller sets of covering test cases." "In some of the comparisons, there also seems to be an implicit assumption that the pairwise tests are representative of the technique generally. We just saw a list of 25 tools. The same web page presents a table showing differences in the numbers of test cases they generate. There is variation here, and perhaps more importantly, there are possible improvements, ..."
The tools list referenced above is at Czerwonka's Pairwise.org site: https://www.pairwise.org/tools.asp Improvements in reducing the number of pairwise test cases are discussed later in Sherwood's review. See also Colbourn's list of best known pairwise templates (covering arrays for t=2). https://www.public.asu.edu/~ccolbou/src/tabby/catable.html
Whether it is easier to generate pairwise or random test cases is dependent on the generator tool of course. In both techniques, an analysis of the system and its operational states is required. Pairwise testing can provide better coverage of pairs with far fewer test cases. Random test cases make sense for stability testing or load testing, when more test cases may be desirable.
Bookworm271 (talk) 17:47, 22 August 2008 (UTC)

Incorrect function for finding the number of tests?

The article states that the number of tests to cover pairwise tests for a test function is the product of the two largest test variable domains (if I am interpreting it correctly). If we consider the case of four binary variables, then the article suggests all pairwise tests can be expressed as 4 vectors. However, this is not true. It requires at least five tests to cover four binary variables (source: https://www.public.asu.edu/~ccolbou/src/tabby/2-2-ca.html). Intuitively, the formula doesn't make sense because it suggests the number of tests does not scale with the number of variables (pick four assignments for one billion binary variables such that all pairs are expressed?)

Glhanes (talk) 19:19, 14 November 2012 (UTC)

It makes perfect sense.

Why would we consider the case of four binary variables? It's trivial and also an exception. The goal is to test every combination of pairs together and so if your options are only pairs, you end-up with more tests than the rule suggests you should.

However, if you'd like to add your exception with the above reference, it wouldn't be removed. --Walter Görlitz (talk) 19:40, 14 November 2012 (UTC)

I don't understand why you think that it's a trivial case. Why wouldn't you want to perform all-pairs testing on four binary variables? You're correct; you want to test every combination of pairs. For four binary variables, this requires five tests (as opposed to 16 in exhaustive testing), and not four as the article's formula suggests. Regardless of the number of variables or the size of the domain of each variable, the formula specified for calculating the minimum number of tests to account for every pairwise combination of assignments is incorrect. The table I linked to clearly shows that the formula is incorrect. The examples it uses are correct, but only coincidentally.Glhanes (talk) 03:40, 26 December 2012 (UTC)

18-11-2019: I would like to add that I was confused too by the statement that if you multiply the two largest variables you will get the number of testcases needed to cover for pairwise testing. When you need to test a lot of binary inputs (PLC digital inputs in my case) you will need more testcases then 2 x 2. I suggest changing/clarifying the article to prevent confusing others. — Preceding unsigned comment added by 131.237.83.2 (talk) 16:32, 18 November 2019 (UTC)

Simplify text

The text is difficult to understand.

Can we avoid words which are not in common use?

E.g. predicate. The entry for predicate is itself confusing, so pointing to it does not help. W102102 (talk) 08:01, 13 May 2022 (UTC)

Remove link to Udemy course

The second paragraph of the introduction contains a link to a paid online course on Udemy, a website which IMO employs highly scammy marketing strategies. I'm not sure if Wikipedia should link to such sites in the article itself, but I'm not too familiar with the rules for this, maybe someone who is can take a look. 195.176.112.243 (talk) 11:46, 14 August 2023 (UTC)