Prioritized Variable-length Test Cases Generation for Finite State Machines

Model-based Testing (MBT) is an effective approach for testing when parts of a system-under-test have the characteristics of a finite state machine (FSM). Despite various strategies in the literature on this topic, little work exists to handle special testing situations. More specifically, when concurrently: (1) the test paths can start and end only in defined states of the FSM, (2) a prioritization mechanism that requires only defined states and transitions of the FSM to be visited by test cases is required, and (3) the test paths must be in a given length range, not necessarily of explicit uniform length. This paper presents a test generation strategy that satisfies all these requirements. A concurrent combination of these requirements is highly practical for real industrial testing. Six variants of possible algorithms to implement this strategy are described. Using a mixture of 180 problem instances from real automotive and defense projects and artificially generated FSMs, all variants are compared with a baseline strategy based on an established N-switch coverage concept modification. Various properties of the generated test paths and their potential to activate fictional defects defined in FSMs are evaluated. The presented strategy outperforms the baseline in most problem configurations. Out of the six analyzed variants, three give the best results even though a universal best performer is hard to identify. Depending on the application of the FSM, the strategy and evaluation presented in this paper are applicable both in testing functional and non-functional software requirements.

READ FULL TEXT VIEW PDF

Authors

page 1

05/20/2020

Alternative Effort-optimal Model-based Strategy for State Machine Testing of IoT Systems

To effectively test parts of the Internet of Things (IoT) systems with a...
05/25/2021

Complete Requirements-based Testing with Finite State Machines

In this paper, new contributions to requirements-based testing with dete...
03/17/2022

Overview of Test Coverage Criteria for Test Case Generation from Finite State Machines Modelled as Directed Graphs

Test Coverage criteria are an essential concept for test engineers when ...
08/20/2020

Model-based Automated Testing of Mobile Applications: An Industrial Case Study

Automatic testing of mobile applications has been a well-researched area...
02/22/2018

Employment of Multiple Algorithms for Optimal Path-based Test Selection Strategy

Executing various sequences of system functions in a system under test r...
02/22/2021

Towards Evidence-based Testability Measurements

Evaluating Software testability can assist software managers in optimizi...
08/22/2019

Automated Generation of Test Models from Semi-Structured Requirements

[Context:] Model-based testing is an instrument for automated generation...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Paper accepted at the 6th International Workshop on Testing Extra-Functional Properties and Quality Characteristics of Software Systems (ITEQS) workshop of the 15th IEEE International Conference on Software Testing, Verification and Validation (ICST) 2022 conference, April 4, 2022 – April 13, 2022, https://icst2022.vrain.upv.es/

Finite State Machine (FSM) character is available in various parts of most of the modern systems, like software, electronics, or the Internet of Things (IoT). To test these systems, a usual approach is to exercise sequences of transitions (test paths) in FSMs to model those parts of the System Under Test (SUT) [5]

. The goal is to create test paths (test path set) in an optimal way, where various criteria can be considered, e.g., minimal test price or maximal probability to detect defects in SUT. The satisfaction of a defined test coverage criteria is a standard prerequisite of such a process

[27, 3]. The Model-based Testing (MBT) approach renders as an effective automated way to achieve this goal [5, 27, 3].

Current literature presents a variety of approaches to generate test paths from an FSM-based SUT model (e.g., [21, 2, 5, 14]). However, fewer works exist for specific cases, in which the following requirements have to be satisfied concurrently: (1) test paths can start and end in defined FSM states, which are a subset of all FSM states, (2) test paths must visit only defined priority states and transitions of FSM; the rest of the states and transitions are not required to be visited by a test path set, and, (3) the length of the created test paths must be in a defined range (not explicitly of a certain uniform length). Concurrent satisfaction of these three requirements gives an FSM-based testing technique good flexibility and potential to serve the needs of industry projects. Formulation of the requirements results from our active discussions with industry testing experts in Rockwell, Siemens, Skoda Auto and Electrolux companies, in which the needs for an FSM-based testing technique were analyzed.

In practical testing, starting and ending a test in certain states of FSM either is costly or not possible [12, 16]. The length of the test paths also plays a significant role. Excessively long test paths might suffer from high maintenance. Additionally, if a test path is interrupted by a defect present in a SUT, it might be challenging to finish the test properly. Opposite extreme, too short test paths might be burdened by overhead that is caused by test initialization clean-up after the test of related managerial efforts [29].

Hence, this paper proposes the Prioritized State Machine Test (PSMT) strategy that generates sets of test paths that satisfy the aforementioned requirements. The strategy employs an FSM-based SUT model and two types of test coverage criteria to model this problem. As a part of the strategy, we present six algorithm variants that generate the test paths to satisfy the given test coverage criteria. As a baseline, we use an alternative N-Switch Reduction algorithm that is based on an established N-switch coverage concept [29, 26, 1]. We compare the results of the presented algorithms using 180 SUT models that are partially derived from real automotive and defense systems, and the parts are generated artificially.

Ii SUT model

The SUT model is a directed multigraph, where is a set of vertices representing FSM states, is a set of edges representing FSM transitions, and is a set of edge labels, edge defined by , where is a start vertex of edge , is an end vertex of edge , and is a label of edge .

Vertex is a start of the state machine, is a set of end vertices of the state machine, is a set of possible starts of test paths, is a set of possible end of test paths, and and can be nonempty. Each edge has defined its priority denoted as as well as each vertex has defined its priority denoted as . This priority is a real number and it holds that and , where and denote minimal and maximal level of priority, .

Test path is a path in that starts in a and ends in a . A test path is a sequence of edges and is a set of all test paths.

Iii Test coverage criteria

We define two test coverage criteria that must be satisfied in . They differ by the number of test path transitions, giving additional flexibility to generate a test path set suitable for a particular case.

satisfies PSMT Basic Coverage, when all the following conditions are satisfied:

  1. Each of test paths must start in a vertex from and end in a vertex from ,

  2. for each , , where is length of a test path in number of its edges,

  3. each vertex and edge with and that can be part of some which starts at some vertex from , ends in some vertex from and , where is length of in number of its edges, must be present in , and,

  4. any path is not a sub-path of other paths in .

On top of these rules, no additional requirements on how and have to be chained or combined in the test paths are given by PSMT Basic Coverage. Further, it is not required that every vertex from must be present as an end vertex of some . To satisfy PSMT Basic Coverage, it is not necessary to visit neither whole nor even .

The PSMT Basic Coverage is suitable for lower intensity tests when the situation does not require to conduct intensive testing or there is a lack of testing resources.

satisfies PSMT Extended Coverage, when all the following conditions are satisfied:

  1. satisfies PSMT Basic Coverage, and,

  2. each edge incoming and outgoing to each vertex with must be present in at least one path , and,

  3. each pair of adjacent edges , in which or , must be part of some .

When satisfies PSMT Extended Coverage, it satisfies also PSMT Basic Coverage criterion. The PSMT Extended Coverage is designed for more intensive tests and more transitions of FSM are executed during the tests. However, to satisfy PSMT Extended Coverage, it is not required to visit all transitions from nor all states .

A consequence of PSMT Basic Coverage and PSMT Extended Coverage is that for a certain combined with certain ranges of and , that satisfies the test coverage criterion might not exist. This situation can be solved by changing and , adding more and to , or by a combination of both.

Iv Algorithms

PSMT strategy is defined by Algorithm 1. It accepts SUT model , , , expected test path length range and , test coverage criterion denotes as that can be PSMT Basic Coverage and PSMT Extended Coverage, and test path set reduction type denoted as that determines the strategy variant. Requirement is a path in or a vertex (a zero length path) that must be present in a . Moreover, each contains at least one . Moreover, must satisfy criteria as defined in Section III for selected .

Algorithm 2 is a subroutine of the PSMT strategy and creates an initial set of test paths that are further reduced by the following algorithms. As the test path set reduction type (specified by ), we analyzed six options, starting with No reduction variant, followed by five reduction variants as defined by separate algorithms 3, 4, 5, 6 and 8.

These algorithms approach the problem of test path set reduction as a well-known Set Cover Problem (SCP), where a set of requirements is an universe and is as a set of subsets of this universe, where each path represents a subset of requirements it covers.

No reduction is a trivial variant that does not reduce .

Function: GenerateTestPathsPSMT
      Input: , , , , , ,
      Output: Set of test paths

1: an empty set of test paths
2: generate requirements for for vertices with and edges with by specification in Section III.
3:for  do
4:      FindShortestPathInRange() Requirement is a path in or vertex
5:     
6: returned by Test Path Reduction algorithm accepting and that is specified by (no reduction or Algorithms 3, 4, 5, 6, 8).
7:return
8:end function
Algorithm 1 Generate for by PSMT strategy

Function: FindShortestPathInRange
      Input: , ,
      Output: Path if no path is found then an empty path is returned

1: and are empty maps of paths. Key in the map is a path length.
2: and are empty queues of paths
3: first and last vertex of
4:push to ;
5:push to
6:while ( is not empty) ( is not empty) do
7:     if  is not empty then
8:          pull from
9:         if  then
10:               first vertex of
11:              if  then
12:                  if  contains a path that can be joined with so that for the result path ,  then
13:                       return concatenated by the path from                   
14:                  if  length of shortest path from  then
15:                       if  does not exist then
16:                                                                                   
17:              if  length of the shortest path from  then
18:                  for  edges incoming to  do
19:                       put ( concatenate ) to                                               
20:     if  is not empty then
21:          pull from
22:         if  then
23:               last vertex of
24:              if  then
25:                  if  contains a path that can be joined with so that for the result path ,  then
26:                       return the path from concatenated by                   
27:                  if  length of shortest path from  then
28:                       if  does not exist then
29:                                                                                   
30:              if  length of the shortest path from  then
31:                  for  edges outgoing from  do
32:                       put ( concatenate ) to                                               
33:return empty path
34:end function
Algorithm 2 Find the shortest path in range

The simplest approach is Random specified by Algorithm 3. It iterates over all until all are covered. If the iterated covers some uncovered requirements, they are marked as covered and the path is added to the final set , otherwise iterating continues. Finally, the reduced set is returned.

Function: Random
      Input: ,
      Output: Reduced set of test paths

1:
2:
3: an empty set of paths
4:while  do
5:      a random path from
6:     
7:      previously uncovered requirements now covered by
8:     if  then
9:         
10:               
11:return
12:end function
Algorithm 3 Random reduction

The variant Sorted is defined in Algorithm 4. The test path set is first sorted in a descendant order by a number of covered requirements. The algorithm continues as in the random variant (see Algorithm 3), but paths are taken in sorted order.

Function: Sorted
      Input: ,
      Output: Reduced set of test paths

1:
2: sorted in descending way by number of covered requirements
3: an empty set of paths
4:
5:while  do
6:      first path from
7:     
8:      requirements covered by
9:     if  then
10:         
11:               
12:return
13:end function
Algorithm 4 Sorted Reduction

Chvatal reduction variant that bases on Chvatal’s algortihm for SCP [10, 31] is defined by Algorithm 5. For performance reasons, the algorithm first filters that are sub-path of other requirements and removes paths that do not cover any . Chvatal’s SCP algorithm uses as a universe and as a set of subsets where each path represents a set of requirements that covers. During this SCP process, the reduced is generated.

Function: Chvatal
      Input: ,
      Output: Reduced set of test paths

1: , where , is not a sub-path of other and
2: from which all test paths that do not contain any are removed
3: reduce using Chvatal’s SCP algorithm as defined in [10] in such a way that is as an universe and is a set of subsets, where each represents a that covers.
4:return
5:end function
Algorithm 5 Chvatal-SCP-based reduction

The next reduction variant, Genetic Algorithm [30] is employed to solve the SCP problem (Algorithm 6). In the selection process, one population individual represents a . Besides and on input, the algorithm employs a set of input configurations (further GA configuration), whose particular values were determined during the experiments as giving the best results. These are: initialProbabilityToSetGene=0.2, probabilityToMutateOneGene=0.4, probabilityToMutateZeroGene=0.6, maxGenerations=100, populationSize=30, maxGenerationsWithoutImprovement=40.

In this variant, we designed our fitness function, which is , where represents a , , and is a count of which are not a sub-path of any . Crossover operation employed in the Genetic Algorithm reduction is further specified in Algorithm 7.

Function: Genetic Algorithm
      Input: , , ,
     
     
     
     
      Output: Reduced set of test paths

1: from which filter requirements that are not a sub-path of any are removed
2:initialize Each individual is represented by an array of bits where each bit is set with probability
3:do
4:      Crossover(, , , , )
5:     
6:      () individuals from having the highest value of the fitness function
7:     
8:      () survivals using a roulette wheel [28] with same GA configuration from
9:     
10:while terminate criteria are not met terminate criteria are met when maximal number of generation is achieved, best possible individual is found or there is no improvement for generations
11: test paths represented by selected best individuals
12:return
13:end function
Algorithm 6 Genetic Algorithm based reduction

Function: Crossover
      Input: ,       ,
     
      Output:

1: init empty set of individuals
2:for  do
3:     Choose and using the roulette wheel using the GA configuration
4:      generate two random points such that
5:     Create and by combination of parents’ chromosomes, such that has zero to gene of , to gene of and to the last gene of . For do the same process, only switch and .
6:     Mutate one random gene of and with probability or if the gene is set or not respectively
7:     
8:return
9:end function
Algorithm 7 Crossover operation of Genetic Algorithm based reduction

The last variant to compare is reduction done via Simulated Annealing as specified by Algorithm 8. In the algorithm, we have used a geometric annealing schedule. Hence, besides and , the algorithm accepts the coefficient for this schedule and, based on the experiment results, was used.

Function: Simulated Annealing
      Input: , , coefficient for geometric annealing schedule
      Output:

1: a random point in the solution space Solution is represented by an array of bits of size . A point defines , that are used in the solution, if bit is set then path of is used. denotes a that is represented by . Further, denotes a such that each is a sub-path of a
2:
3:
4:do
5:     
6:     do
7:          with a random bit switched
8:         
9:         if  then
10:              
11:              if  then Check whether the new solution is the new best one
12:                                 
13:         else if  then returns the natural exponential of x and returns random number

of uniform distribution

14:                        
15:     while the loop is iterated -times where is actual temperature Equilibrium is reached
16:      Geometric cooling schedule is used
17:while  Is frozen
18:
19:return
20:end function
Algorithm 8 Simulated Annealing reduction

The N-Switch Reduction strategy bases on established N-Switch Coverage concept used in FSM testing[29, 26, 1] and is defined by Algorithm 9. The algorithm accepts the same inputs as the PSMT strategy (see Algorithm 1) and produces as an output. The algorithm first generates all paths in of length , where , which are put to the initial . Then, the algorithm reduces to keep only paths that starts in and ends in some vertex from . Then, paths of are further analyzed if more paths start in a particular vertex from and if so, only one of these paths is kept in .

Function: N-Switch Reduction
      Input: , , , , ,
      Output: Set of test paths

1:; empty set of paths
2:for  do
3:      all paths in such that for each and is the first edge in
4:     
5: generate requirements for for vertices with and edges with by specification in Section III.
6:for each  do
7:      requirements from which are sub-path of
8:     if  then
9:         ;
10:         if  then
11:              break the loop               
12:return
13:end function
Algorithm 9 Generate test paths for SUT model by N-Switch Reduction strategy

V Experiments

All presented algorithms are implemented in the developer version of the Oxygen111http://still.felk.cvut.cz/oxygen/ platform [9, 7]. The platform allows for the creation of the SUT model in its graphical editor. It also allows to import and export of the SUT models in open XML-based formats. The selected algorithms can generate test path sets for a set of SUT models. Test paths can then visualized in the model and exported. An example of Oxygen user interface with visualized test path in a SUT model is given in Figure 1. SUT model states from , and are highlighted by green, red and orange color, respectively. The developer version of the platform allows for batch execution of the present algorithms on multiple SUT models and export of summary results for further analysis and evaluation.

Fig. 1: Example of Oxygen editor with visualization of a test path in a SUT model.

V-a Experiment set-up and method

For the experiments, we used 180 problem instances . These instances are composed of: (1) state machine models of real industrial project, (2) modifications of the industrial real models, and (3) problem instances generated artificially. Regarding the industrial FSMs, six models describing various parts of tested Skoda cars were used. The Skoda Auto testing team created these models in a special Skoda version of the Oxygen tool supporting . These models were further modified by adding cycles to an FSM, adding possible test starts and test ends, adding or removing a state, and adding and removing a transition. In this paper, by a cycle, we mean a configuration of edges that allow non-empty trails in which only the first and last vertices are equal. The result was 24 problem instances. Another 12 problem instances were created by the same method for FSMs for data transmission protocols and mission management in the Digital Triage Assistant project222https://www.natomultimedia.tv/app/asset/656263, a joint project of CTU in Prague, Johns Hopkins University, University of Defence, NATO ACT IH and other partners. Non-disclosure agreements and confidentiality restrictions of both projects allow making publicly available only abstracted topology of FSMs without the names of states and transitions.

To create enough variety of possible FSMs, we generated an additional set of 144 problem instances by a specialized tool, developed as a master thesis by Richard Sadlon333https://dspace.cvut.cz/handle/10467/97079, CTU in Prague. The tool generates problem instances by expected properties of the graph entered as an input. In this process, , , number of cycles, , , , and can be specified.

Table I presents the selected properties of all problem instances used in the experiments. In Table I, denotes number of cycles, denotes average length of these cycles, denotes number of groups of parallel edges present in (in these groups, edges start and end in the same vertex), denotes total number of parallel edges in , denotes average node incoming degree, denotes average node outgoing degree, and denotes average node degree. Further, denotes the state in which a test path can both start and end.

Regarding the priorities defined in the SUT model, as well as were set to range 0 to 3 for all , resp. , for all problem instances. Then, was set to 2 and was set to 3 in a unified way for the experiment. In Table I, denotes number of for which and I, denotes number of for which .

To evaluate the effectiveness of the generated to detect defects, the problem instances were extended by fictional defects of two types. Type 1 Defect is present at an edge representing FSM transition and is considered to be activated when a visits . In Table I, number of these defects in problem instances is denoted as .

Type 2 defect is present at two edges and , where there is a path in from to . To consider Type 2 defect to be activated, a visits and then visits . Type 2 Defects simulate data consistency defects, when a transition induces an internal inconsistency to a SUT and other transition () causes a defective behavior of the SUT.

In Table I, the number of Type 2 defects in problem instances is denoted as and average distance between and in number of edges is denoted as .

property Mean Median MIN MAX
20.33 15 10 57
34.81 35 19 95
3.23 3 0 18
9.27 8 0 31
2.09 1 1 21
0.53 0 0 18
0.27 0 0 9
1.77 1.53 1.04 2.33
1.77 1.53 1.04 2.33
3.55 3.07 2.08 4.67
2.33 2 1 17
2.59 2 1 25
1.14 1 0 6
5.66 5 0 21
9.75 9 2 34
7.00 6 1 28
6.26 6 1 27
2.69 2.50 0.67 6.60
TABLE I: Properties of problem instances used in the experiments.

We evaluated the following properties of : , total length of all in number of edges (denoted as ), average length of all in number of edges (denoted as ), number of unique edges in all (denoted as ). We also analyzed that expresses how many non-unique FSM transitions ( edges) need to repeat in a test path to test the unique transitions. Higher indicates higher ”edge duplication” in .

To evaluate the defect detection potential of , we analyzed the number of simulated defects of Type 1 and Type 2 that were activated by a , denoted as and , respectively. The last property to evaluate is averaged number of simulated defects activated by one test path step, defined as and for Type 1 and Type 2 simulated defects, respectively.

We ran all presented algorithms for both PSMT Basic Coverage and PSMT Extended Coverage criteria. In each of these criteria, we present and discuss the results for two test path length ranges: and . If no was returned for a particular problem instance for the individual configuration of test path length range and the coverage criterion by any of the compared algorithms, the result record was not taken into account for all algorithms for this problem instance. Because of the non-deterministic nature of the Genetic Algorithm and Simulated Annealing test path reduction variants of the PSMT strategy, we ran all computations three times and averaged the results.

V-B Results

Table II presents the averaged results for all problem instances for all compared algorithms and both coverage criteria for the test path length range . The major properties we further analyze and discuss are , and