DeepAI
Log In Sign Up

Constrained locating arrays for combinatorial interaction testing

12/06/2017
by   Hao Jin, et al.
0

This paper introduces the notion of Constrained Locating Arrays (CLAs), mathematical objects which can be used for fault localization in a testing process for information systems. CLAs extend ordinary locating arrays to make them applicable to testing of systems that have constraints on test parameters. Such constraints are common in real-world systems; thus CLA enhances the applicability of locating arrays to practical testing problems.

READ FULL TEXT VIEW PDF

page 1

page 2

page 3

page 4

10/13/2021

Constrained Detecting Arrays: Mathematical Structures for Fault Identification in Combinatorial Interaction Testing

Context: Detecting arrays are mathematical structures aimed at fault ide...
08/07/2019

A Survey of Constrained Combinatorial Testing

Combinatorial Testing (CT) is a potentially powerful testing technique, ...
05/26/2021

Incomplete MaxSAT Approaches for Combinatorial Testing

We present a Satisfiability (SAT)-based approach for building Mixed Cove...
10/22/2020

Position-Agnostic Multi-Microphone Speech Dereverberation

Neural networks (NNs) have been widely applied in speech processing task...
11/10/2016

XCSP3: An Integrated Format for Benchmarking Combinatorial Constrained Problems

We propose a major revision of the format XCSP 2.1, called XCSP3, to bui...
04/16/2019

Finding minimum locating arrays using a CSP solver

Combinatorial interaction testing is an efficient software testing strat...
12/01/2013

A Combined Approach for Constraints over Finite Domains and Arrays

Arrays are ubiquitous in the context of software verification. However, ...

1 Introduction

Combinatorial interaction testing is a well-known strategy for software testing. In the strategy, a System Under Test (SUT) is modeled as a finite set of factors and every interaction of interest is exercised by at least one test. Empirical results suggest that testing interactions involving a fairly small number of factors, typically two or three, suffices to reveal most of latent faults. Many studies have been developed to construct a small test set for combinatorial interaction testing. Such test sets are often called Covering Arrays (CAs). Surveys on these studies can be found in, for example, [4, 9, 15].

An important direction of extending the capability of combinatorial interaction testing is to add fault localization capability to it. An enabling technique for this is using a Locating Array (LA) as a test suite [6]. In [6] LAs of a few different types are defined. For example, a -LA enables to locate a set of failure-triggering -way interactions using the test outcome.

The purpose of this paper is to extend the notion of LAs to expand the applicability to practical testing problems. Specifically, we propose Constrained Locating Arrays (CLAs) which can be used to detect failure-triggering interactions in the presence of constraints. Constraints, which prohibit some particular tests, are common in real-world systems. Constraint handling in combinatorial interaction testing has been well studied [1]. The main focus of the previous studies is on constructing a test set, often called a Constrained Covering Array (CCA), that consists only of constraint-satisfying tests and covers all interactions that can occur in constraint-satisfying tests.

CLAs requires additional considerations about constraints. Specifically, constraints may make it impossible to distinguish a failure-triggering interaction or set of such interactions from another; hence a special treatment must be needed to deal with such an inherently indistinguishable pair. By extending LAs with the concept of distinguishability, we provide the formal definition of CLAs. We also show some non-trivial examples of CLAs.

The rest of the paper is organized as follows. Section 2 describes the SUT model and the definition of locating arrays, as well as some related notions. Section 3 presents the definition of CLAs and some basic theorems about them. Section 4 presents a few examples of CLAs together with a conceptual description of a possible approach of mechanical CLA constructions. Section 5 briefly summarizes related work. Section 6 concludes the paper with possible future directions of work.

2 Preliminaries

factors : Display : Email : Camera : Video : Video
Viewer Camera Ringtones
values 0 : 16 MC 0 : Graphical 0 : 2 MP 0 : Yes 0 : Yes
1 : 8 MC 1 : Text 1 : 1 MP 1 : No 1 : No
2 : BW 2 : None 2 : None
constraints
Graphical email viewer requires color display
2 Megapixel camera requires color display
 Graphical email viewer not supported with 2 Megapixel camera
8 Million color display does not support 2 Megapixel camera
Video camera requires camera and color display
Video ringtones cannot occur with No video camera
The combination of 16 Million colors, Text email
viewer and 2 Megapixel camera will not be supported
Figure 1: Example of an SUT [3].

An SUT is modeled as where is a set of factors, is a set of domains for the factors, and is a mapping that represents constraints. Each domain consists of two or more consecutive integers ranging from 0; i.e., (). A test is an element of . A test is valid if and only if (iff) it satisfies the constraints , i.e., . For a set of factors, , the set such that for all is a -way interaction or an interaction of strength . Hence a test contains (covers) -way interactions. (The only 0-way interaction is .) Note that a -way interaction and a test can be treated interchangeably. Thus we write iff a test covers an interaction . An interaction is valid iff for some valid test .

As a running example, consider a classic cell-phone example taken from [3] (Fig. 1). This SUT model has five factors which have three or two values in their domains. The constraints consist of seven parts. Test , for example, is valid, whereas test is not valid (invalid) because it violates the third and fourth constraints. Similarly, two-way interaction is valid, since it occurs in valid test . On the other hand, is invalid, since it violates constraint .

A test suite is a (possibly empty) collection of tests and thus can be represented as an array when the number of tests is . For such an array and interaction , we let denote the set of tests (rows) of in which the interaction is covered. For a set of interactions , we define .

An interaction is either failure-triggering or not. The result of executing a test is fail iff covers at least one failure-triggering interaction; otherwise the result is pass. Hence the result of executing a test suite

is a vector of size

, each element being either pass or fail.

When there are no constraints, i.e., for any , a CA can be used to detect the existence of fault-triggering interactions of a given strength or less. Let be the set of all -way interactions. Formally, a -CA is defined by the following condition:

-CA :

On the other hand, an LA can be used as a test suite that locates the set of failure-triggering interactions. Colbourn and McClary introduced a total of six types of LAs in [6]. The definitions of the two most basic types of LAs are shown below.

-LA such that :
-LA such that , :

The definition of other two types of LAs, namely -LAs and -LAs, requires the notion of independence [6]. Let be the set of all interactions of strength at most , i.e., . A set of interactions (interaction set) is independent iff there do not exist two interactions with . For example, consider a set of two interactions for the running example. This interaction set is not independent because . Note that if two interactions are both failure-triggering and , then the failure triggered by always masks the failure triggered by . Because of this, it is natural to limit the scope of fault localization to independent interaction sets. Based on and the notion of independent interaction sets, the two types of LAs are defined as follows.

-LA such that and are independent:
-LA such that , and are independent:

We do not consider the remaining two types of locating arrays, namely -LAs and -LAs, because they either exist in trivial cases or otherwise are equivalent to - and -LAs.

Figure 2 shows a -LA for the running example shown in Fig. 1. Let be the LA and be the th row. If the pass/fail result were obtained for all these tests, any failure-triggering single two-way interaction could be identified. For example, if only the first test failed, then the failure-triggering interaction would be determined to be , because holds only for , provided that and for . However, this array cannot be used for testing the system because of the constraints. For example, is not valid and thus cannot be executed in reality.

0 0 0 0 0
0 0 1 1 1
0 0 2 0 1
0 1 1 0 0
0 2 0 0 1
1 0 1 0 1
1 0 2 1 1
1 1 0 0 1
1 1 2 1 0
1 2 0 1 0
2 0 2 0 0
2 1 1 1 1
2 2 0 1 0
2 2 1 0 0
2 2 2 1 1
Figure 2: (1,2)-LA for the running example. Constraints are not taken into account.

3 Constrained locating arrays

In the presence of constrains, a test suite must consist of only valid tests. Clearly, this means that invalid interactions cannot and need not be located. From now on, we assume that an array representing a test suite consists of only valid tests or no rows.

In the field of practical software testing, this problem has been circumvented by, instead of CAs, using Constrained Covering Arrays (CCAs). Let be the set of all valid -way interactions. Then a CCA of strength , denoted as -CCA, is defined as follows.

-CCA :


In words, a -CCA is an array that covers all valid interactions of strength .

When incorporating constraints into LA, it is crucial to take into consideration, in addition to the presence of invalid interactions, the fact that constraints may make it impossible to identify some set of failure-triggering interactions, which could be identified if no constraints existed. This requires us the notion of distinguishability to formally define CLAs.

Definition 1

A pair of sets of valid interactions, and , are distinguishable iff for some array consisting of valid tests.

For the running example, are not distinguishable (indistinguishable), since any valid test contains either both of the two-way interactions or none of them. That is, tests that cover exactly one of the two interaction sets (e.g., (0 1 0 0 0) or (1 2 0 0 0)) are all invalid. Hence no array exists such that .

It should be noted that even if there are no constraints, there can be some indistinguishable pairs of interaction sets. In the running example, two interaction sets , are indistinguishable even if the constraints were removed, because any test has 0 or 1 on factors and . Another extreme case is when and are identical. Clearly, identical interactions are always indistinguishable.

Definition 2

Let and . Let be the set of all valid -way interactions and be the set of all valid interactions of strength at most . An array that consists of valid tests or no rows is a -, -, - or -CLA iff the corresponding condition shown below holds.

-CLA such that and are distinguishable:
-CLA such that , and are distinguishable:
-CLA such that and are independent and distinguishable:
-CLA such that , and are independent and distinguishable:

(In extreme cases where no two such interaction sets exist, any is a CLA.)

The intuition of the definition is that if the SUT has a set of (or ) failure-triggering interactions, then the test outcome obtained by executing all tests in will be different from the one that would be obtained when the SUT had a different set of (or ) failure-triggering interactions, unless the two interaction sets are not distinguishable.

The following theorem follows from the definition.

Theorem 1

A -CLA is a - and -CLA. A -CLA and a -CLA are both a -CLA. A -CLA and a -CLA are a -CLA and a -CLA, respectively. A -CLA and a -CLA are a -CLA and a -CLA, respectively.

Theorem 2 states that when there are no constraints, an LA, if existing, and a CLA are equivalent.

Theorem 2

Suppose that the SUT has no constraints, i.e., for all . If an LA exists, then 1) is a CLA with the same parameters, and 2) any CLA with the same parameters as is an LA (which is possibly different from ) with the same parameters.


Proof. Suppose that is a -LA. Let be any two interaction sets such that . 1) If , then . If , then they are not distinguishable. Hence is a -CLA. 2) Suppose that an array is a -CLA. If , then and thus they are distinguishable, which in turn implies . If , then they are not distinguishable and trivially . Hence is a -LA. The same argument applies to the other three types of LAs.

It should be noted that a CLA always exists whether there are constraints or not, as will be shown in Theorem 3. On the other hand, LAs do not always exist. For example, no -LAs exist for the running example: Consider and . Then and both include all rows; thus for any . The guarantee of the existence of CLAs comes from the definition which exempts indistinguishable pairs of interaction sets from fault localization. In that sense, CLAs can be viewed as a “best effort” variant of LAs.

Lemma 1

A pair of sets of valid interactions, and , are distinguishable iff there is a valid test that covers some interaction in or but no interactions in or , respectively, i.e., for some valid test , or .


Proof. (If part) Suppose that there is such a valid test . Consider an array that contains . Then, either or ; thus . (Only if part) Suppose that there is no such valid test, i.e., for every valid test , and . This means that for every valid test , or . Hence for any test in , or . As a result, for any , .

Theorem 3

If is an array consisting of all valid tests, then is a CLA with any parameters.


Proof. Let and be any interaction sets that are distinguishable. By Lemma 1, a valid test exists such that or . Since contains this test and by the same argument of the proof of the if-part of Lemma 1, .

Although Theorem 3 guarantees that a test suite consisting of all valid tests is a CLA, it is desirable to use a smaller test suite in practice. In the next section, we show some CLAs for the running SUT example, which are the smallest we have obtained for particular parameters.

4 Examples of CLAs

In this section, we show (1,1)-, -, - and -CLAs for the running SUT example. Figures 3, 4, 5 and 6 respectively show these CLAs. The sizes (i.e., the number of rows) of these arrays are 5, 12, 15 and 28. The number of valid tests for the running example is 31; thus these CLAs, except the -CLA, are considerably smaller than the array that consists of all valid tests. On the other hand, the -CLA is almost as large as the exhaustive one. The three missing valid tests are (0, 2, 1, 0, 1), (0, 2, 1, 1, 1) and (1, 1, 1, 1, 1).

In the remaining part of the section, we briefly describe how to obtain such CLAs. In the first step, all valid interactions of strength or , i.e., or , are enumerated. This can be done by enumerating all valid tests and then enumerating all interactions of strength or that are contained in at least one of the valid tests. For the running example, all interactions of strength are valid, except ten two-way interactions listed below.

In the second step, all distinguishable pairs of independent sets of valid interactions of size  or are enumerated. Given a pair of sets of valid interactions, their distinguishability can be determined by checking if there is a valid test that covers an interaction in one set and none of the interactions in the other set (Lemma 1). For the running example, all pairs such that are distinguishable. That is, any pair of distinct one-way interactions are distinguishable. Figure 7 shows pairs of interaction sets that are not distinguishable for the other parameters .

Finally, an array consisting of valid tests is somehow created as a candidate for a CLA and then checked to see if the array meets the condition of a CLA. If the array is not a CLA, then a new array is repeatedly created and checked until a CLA is found.

In CCA generation algorithms, the repeated creation of candidate arrays is often performed by meta-heuristic algorithms, such as simulated annealing 

[8] or tabu search [11], or by systematic search based on combinatorial problem solvers, such as Boolean satisfiability (SAT) solvers [14, 20]. We believe that both approaches can also be used to generate CLAs. Indeed we obtained the two CLAs with the aid of a combinatorial problem solver. Although the details of how we used it are beyond the scope of the paper, we plan to report them in future.

0 0 1 1 1
0 2 0 0 1
1 1 1 0 0
1 2 2 1 1
2 1 1 1 1
Figure 3: (1,1)-CLA for the running example.
0 0 1 0 0
0 0 2 1 1
0 1 1 0 0
0 1 2 1 1
0 2 0 0 0
0 2 0 0 1
0 2 0 1 1
1 0 1 0 1
1 2 1 0 0
1 2 2 1 1
2 1 1 1 1
2 2 2 1 1
Figure 4: -CLA for the running example.
0 0 1 0 0
0 0 2 1 1
0 1 1 0 1
0 2 0 0 0
0 2 0 0 1
0 2 0 1 1
0 2 2 1 1
1 0 1 0 1
1 0 1 1 1
1 1 1 0 0
1 1 2 1 1
1 2 1 0 0
2 1 1 1 1
2 1 2 1 1
2 2 1 1 1
Figure 5: -CLA for the running example.
0 0 1 0 0
0 0 1 0 1
0 0 1 1 1
0 0 2 1 1
0 1 1 0 0
0 1 1 0 1
0 1 1 1 1
0 1 2 1 1
0 2 0 0 0
0 2 0 0 1
0 2 0 1 1
0 2 1 0 0
0 2 2 1 1
1 0 1 0 0
1 0 1 0 1
1 0 1 1 1
1 0 2 1 1
1 1 1 0 0
1 1 1 0 1
1 1 2 1 1
1 2 1 0 0
1 2 1 0 1
1 2 1 1 1
1 2 2 1 1
2 1 1 1 1
2 1 2 1 1
2 2 1 1 1
2 2 2 1 1
Figure 6: -CLA for the running example.
{{(1, 1)}}, {{(1, 1)}, {(3, 1)}} {{(2, 3)}}, {{(2, 3)}, {(3, 1)}}
{{(4, 1)}}, {{(4, 1)}, {(5, 1)}} {{(4, 2)}}, {{(1, 3)}, {(4, 2)}}
{{(4, 2)}}, {{(3, 3)}, {(4, 2)}} {{(5, 2)}}, {{(1, 3)}, {(5, 2)}}
{{(5, 2)}}, {{(3, 3)}, {(5, 2)}} {{(5, 2)}}, {{(4, 2)}, {(5, 2)}}
{{(1, 3)}, {(4, 2)}}, {{(3, 3)}, {(4, 2)}} {{(1, 3)}, {(5, 2)}}, {{(3, 3)}, {(5, 2)}}
{{(1, 3)}, {(5, 2)}}, {{(4, 2)}, {(5, 2)}} {{(3, 3)}, {(5, 2)}}, {{(4, 2)}, {(5, 2)}}
{{(4, 1)}, {(4, 2)}}, {{(4, 1)}, {(5, 2)}} {{(4, 1)}, {(4, 2)}}, {{(5, 1)}, {(5, 2)}}
{{(4, 1)}, {(5, 2)}}, {{(5, 1)}, {(5, 2)}}

(a) such that and and are indistinguishable.



{{(1, 0), (3, 0)}}, {{(2, 2), (3, 0)}} {{(1, 2), (4, 1)}}, {{(1, 2), (5, 1)}}
{{(3, 2), (4, 1)}}, {{(3, 2), (5, 1)}}

(b) such that and and are indistinguishable. (Since , are trivially independent.)



{ {(1,0)} }, { {(1,0)}, {(3,0)} }
{ {(1,2), (4,1)} }, { {(1,2), (4,1)}, {(1,2), (5,1)} }
{ {(1,0), (2,0)}, {(1,1), (2,0)} }, { {(1,2), (4,1)}, {(1,2), (5,1)} }

(c) Some examples of such that and and are independent and indistinguishable.

Figure 7: Indistinguishable pairs of sets of interactions

5 Related Work

Constraint handling has been an important issue in combinatorial interaction testing, even before the name of this testing approach was coined. Early work includes, for example, [19, 2]. A recent systematic literature review on constrained interaction testing listed 103 research papers addressing this particular problem [1].

In contrast, research on LAs is still in the early stage [7]. Mathematical properties and constructions of LAs are presented in [17, 18, 5], but only for some specific cases. Computational constructions of LAs are proposed in [13, 10]. None of these previous studies consider constraints. Mathematical objects similar to LAs include Detecting Arrays [6, 16] and Error Locating Arrays [12]. To our knowledge, no attempts have been reported to incorporate constraints into these arrays, either.

6 Conclusions

In this paper, we introduced the notion of Constrained Locating Arrays (CLA), which generalize locating arrays by incorporating constraints on test parameters into them. The extension enables to apply locating arrays to testing of real-world systems which usually have such constraints. We proved some basic properties of CLAs and showed a few nontrivial examples of CLAs. Possible future research directions include, for example, devising systematic constructions of CLAs and investigating the usefulness of CLAs in real-world testing.

Acknowledgements

This work was supported by JSPS KAKENHI Grant Number JP15K00098.

References

  • [1] Ahmed, B.S., Zamli, K.Z., Afzal, W., Bures, M.: Constrained interaction testing: A systematic literature study. IEEE Access 5(99), 1–1 (2017). DOI 10.1109/ACCESS.2017.2771562
  • [2] Cohen, D.M., Dalal, S.R., Fredman, M.L., Patton, G.C.: The AETG system: An approach to testing based on combinatorial design. IEEE Trans. on Software Engineering 23(7), 437–444 (1997). DOI http://dx.doi.org/10.1109/32.605761
  • [3] Cohen, M.B., Dwyer, M.B., Shi, J.: Constructing interaction test suites for highly-configurable systems in the presence of constraints: A greedy approach. IEEE Trans. on Software Engineering 34, 633–650 (2008). DOI 10.1109/TSE.2008.50
  • [4] Colbourn, C.J.: Combinatorial aspects of covering arrays. Le Matematiche 58, 121–167 (2004)
  • [5] Colbourn, C.J., Fan, B., Horsley, D.: Disjoint spread systems and fault location. SIAM Journal on Discrete Mathematics 30(4), 2011–2026 (2016). DOI 10.1137/16M1056390
  • [6] Colbourn, C.J., McClary, D.W.: Locating and detecting arrays for interaction faults.

    Journal of Combinatorial Optimization

    15(1), 17–48 (2008).
    DOI 10.1007/s10878-007-9082-4
  • [7] Colbourn, C.J., Syrotiuk, V.R.: Coverage, location, detection, and measurement. In: 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 19–25 (2016). DOI 10.1109/ICSTW.2016.38
  • [8] Garvin, B.J., Cohen, M.B., Dwyer, M.B.: Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empirical Software Engineering 16(1), 61–102 (2011). DOI 10.1007/s10664-010-9135-7
  • [9] Grindal, M., Offutt, J., Andler, S.F.: Combination testing strategies: A survey. Software Testing, Verification and Reliability 15(3), 167–199 (2005). DOI 10.1002/stvr.319
  • [10] Konishi, T., Kojima, H., Nakagawa, H., Tsuchiya, T.: Finding minimum locating arrays using a sat solver. In: 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 276–277 (2017). DOI 10.1109/ICSTW.2017.49
  • [11] Lin, J., Luo, C., Cai, S., Su, K., Hao, D., Zhang, L.: TCA: An efficient two-mode meta-heuristic algorithm for combinatorial test generation (t). In: 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 494–505 (2015). DOI 10.1109/ASE.2015.61
  • [12] Martínez, C., Moura, L., Panario, D., Stevens, B.: Locating errors using ELAs, covering arrays, and adaptive testing algorithms. SIAM Journal on Discrete Mathematics 23(4), 1776–1799 (2010). DOI 10.1137/080730706
  • [13] Nagamoto, T., Kojima, H., Nakagawa, H., Tsuchiya, T.: Locating a faulty interaction in pair-wise testing. In: Proc. IEEE 20th Pacific Rim International Symposium on Dependable Computing (PRDC 2014), pp. 155–156 (2014). DOI 10.1109/PRDC.2014.26
  • [14] Nanba, T., Tsuchiya, T., Kikuno, T.: Using satisfiability solving for pairwise testing in the presence of constraints. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences 95-A(9), 1501–1505 (2012). DOI 10.1587/transfun.E95.A.1501
  • [15] Nie, C., Leung, H.: A survey of combinatorial testing. ACM Computing Surveys 43, 11:1–11:29 (2011). DOI 10.1145/1883612.1883618
  • [16] Shi, C., Tang, Y., Yin, J.: The equivalence between optimal detecting arrays and super-simple OAs. Designs, Codes and Cryptography 62(2), 131–142 (2012). DOI 10.1007/s10623-011-9498-9
  • [17] Shi, C., Tang, Y., Yin, J.: Optimal locating arrays for at most two faults. Science China Mathematics 55(1), 197–206 (2012). DOI 10.1007/s11425-011-4307-5
  • [18] Tang, Y., Colbourn, C.J., Yin, J.: Optimality and constructions of locating arrays. Journal of Statistical Theory and Practice 6(1), 20–29 (2012). DOI 10.1080/15598608.2012.647484
  • [19] Tatsumi, K.: Test case design support system. In: Proc. of International Conference on Quality Control (ICQC’87), pp. 615–620 (1987)
  • [20] Yamada, A., Kitamura, T., Artho, C., Choi, E., Oiwa, Y., Biere, A.: Optimization of combinatorial testing by incremental SAT solving. In: 8th IEEE International Conference on Software Testing, Verification and Validation, ICST 2015, Graz, Austria, April 13-17, 2015, pp. 1–10 (2015). DOI 10.1109/ICST.2015.7102599