Paper accepted at the 6th International Workshop on Testing Extra-Functional Properties and Quality Characteristics of Software Systems (ITEQS) workshop of the 15th IEEE International Conference on Software Testing, Verification and Validation (ICST) 2022 conference, April 4, 2022 – April 13, 2022, https://icst2022.vrain.upv.es/
Finite State Machine (FSM) character is available in various parts of most of the modern systems, like software, electronics, or the Internet of Things (IoT). To test these systems, a usual approach is to exercise sequences of transitions (test paths) in FSMs to model those parts of the System Under Test (SUT)  . The goal is to create test paths (test path set) in an optimal way, where various criteria can be considered, e.g., minimal test price or maximal probability to detect defects in SUT. The satisfaction of a defined test coverage criteria is a standard prerequisite of such a process
. The goal is to create test paths (test path set) in an optimal way, where various criteria can be considered, e.g., minimal test price or maximal probability to detect defects in SUT. The satisfaction of a defined test coverage criteria is a standard prerequisite of such a process[27, 3]. The Model-based Testing (MBT) approach renders as an effective automated way to achieve this goal [5, 27, 3].
Current literature presents a variety of approaches to generate test paths from an FSM-based SUT model (e.g., [21, 2, 5, 14]). However, fewer works exist for specific cases, in which the following requirements have to be satisfied concurrently: (1) test paths can start and end in defined FSM states, which are a subset of all FSM states, (2) test paths must visit only defined priority states and transitions of FSM; the rest of the states and transitions are not required to be visited by a test path set, and, (3) the length of the created test paths must be in a defined range (not explicitly of a certain uniform length). Concurrent satisfaction of these three requirements gives an FSM-based testing technique good flexibility and potential to serve the needs of industry projects. Formulation of the requirements results from our active discussions with industry testing experts in Rockwell, Siemens, Skoda Auto and Electrolux companies, in which the needs for an FSM-based testing technique were analyzed.
In practical testing, starting and ending a test in certain states of FSM either is costly or not possible [12, 16]. The length of the test paths also plays a significant role. Excessively long test paths might suffer from high maintenance. Additionally, if a test path is interrupted by a defect present in a SUT, it might be challenging to finish the test properly. Opposite extreme, too short test paths might be burdened by overhead that is caused by test initialization clean-up after the test of related managerial efforts .
Hence, this paper proposes the Prioritized State Machine Test (PSMT) strategy that generates sets of test paths that satisfy the aforementioned requirements. The strategy employs an FSM-based SUT model and two types of test coverage criteria to model this problem. As a part of the strategy, we present six algorithm variants that generate the test paths to satisfy the given test coverage criteria. As a baseline, we use an alternative N-Switch Reduction algorithm that is based on an established N-switch coverage concept [29, 26, 1]. We compare the results of the presented algorithms using 180 SUT models that are partially derived from real automotive and defense systems, and the parts are generated artificially.
Ii SUT model
The SUT model is a directed multigraph, where is a set of vertices representing FSM states, is a set of edges representing FSM transitions, and is a set of edge labels, edge defined by , where is a start vertex of edge , is an end vertex of edge , and is a label of edge .
Vertex is a start of the state machine, is a set of end vertices of the state machine, is a set of possible starts of test paths, is a set of possible end of test paths, and and can be nonempty. Each edge has defined its priority denoted as as well as each vertex has defined its priority denoted as . This priority is a real number and it holds that and , where and denote minimal and maximal level of priority, .
Test path is a path in that starts in a and ends in a . A test path is a sequence of edges and is a set of all test paths.
Iii Test coverage criteria
We define two test coverage criteria that must be satisfied in . They differ by the number of test path transitions, giving additional flexibility to generate a test path set suitable for a particular case.
satisfies PSMT Basic Coverage, when all the following conditions are satisfied:
Each of test paths must start in a vertex from and end in a vertex from ,
for each , , where is length of a test path in number of its edges,
each vertex and edge with and that can be part of some which starts at some vertex from , ends in some vertex from and , where is length of in number of its edges, must be present in , and,
any path is not a sub-path of other paths in .
On top of these rules, no additional requirements on how and have to be chained or combined in the test paths are given by PSMT Basic Coverage. Further, it is not required that every vertex from must be present as an end vertex of some . To satisfy PSMT Basic Coverage, it is not necessary to visit neither whole nor even .
The PSMT Basic Coverage is suitable for lower intensity tests when the situation does not require to conduct intensive testing or there is a lack of testing resources.
satisfies PSMT Extended Coverage, when all the following conditions are satisfied:
satisfies PSMT Basic Coverage, and,
each edge incoming and outgoing to each vertex with must be present in at least one path , and,
each pair of adjacent edges , in which or , must be part of some .
When satisfies PSMT Extended Coverage, it satisfies also PSMT Basic Coverage criterion. The PSMT Extended Coverage is designed for more intensive tests and more transitions of FSM are executed during the tests. However, to satisfy PSMT Extended Coverage, it is not required to visit all transitions from nor all states .
A consequence of PSMT Basic Coverage and PSMT Extended Coverage is that for a certain combined with certain ranges of and , that satisfies the test coverage criterion might not exist. This situation can be solved by changing and , adding more and to , or by a combination of both.
PSMT strategy is defined by Algorithm 1. It accepts SUT model , , , expected test path length range and , test coverage criterion denotes as that can be PSMT Basic Coverage and PSMT Extended Coverage, and test path set reduction type denoted as that determines the strategy variant. Requirement is a path in or a vertex (a zero length path) that must be present in a . Moreover, each contains at least one . Moreover, must satisfy criteria as defined in Section III for selected .
Algorithm 2 is a subroutine of the PSMT strategy and creates an initial set of test paths that are further reduced by the following algorithms. As the test path set reduction type (specified by ), we analyzed six options, starting with No reduction variant, followed by five reduction variants as defined by separate algorithms 3, 4, 5, 6 and 8.
These algorithms approach the problem of test path set reduction as a well-known Set Cover Problem (SCP), where a set of requirements is an universe and is as a set of subsets of this universe, where each path represents a subset of requirements it covers.
No reduction is a trivial variant that does not reduce .
The simplest approach is Random specified by Algorithm 3. It iterates over all until all are covered. If the iterated covers some uncovered requirements, they are marked as covered and the path is added to the final set , otherwise iterating continues. Finally, the reduced set is returned.
The variant Sorted is defined in Algorithm 4. The test path set is first sorted in a descendant order by a number of covered requirements. The algorithm continues as in the random variant (see Algorithm 3), but paths are taken in sorted order.
Chvatal reduction variant that bases on Chvatal’s algortihm for SCP [10, 31] is defined by Algorithm 5. For performance reasons, the algorithm first filters that are sub-path of other requirements and removes paths that do not cover any . Chvatal’s SCP algorithm uses as a universe and as a set of subsets where each path represents a set of requirements that covers. During this SCP process, the reduced is generated.
The next reduction variant, Genetic Algorithm  is employed to solve the SCP problem (Algorithm 6). In the selection process, one population individual represents a . Besides and on input, the algorithm employs a set of input configurations (further GA configuration), whose particular values were determined during the experiments as giving the best results. These are: initialProbabilityToSetGene=0.2, probabilityToMutateOneGene=0.4, probabilityToMutateZeroGene=0.6, maxGenerations=100, populationSize=30, maxGenerationsWithoutImprovement=40.
In this variant, we designed our fitness function, which is , where represents a , , and is a count of which are not a sub-path of any . Crossover operation employed in the Genetic Algorithm reduction is further specified in Algorithm 7.
The last variant to compare is reduction done via Simulated Annealing as specified by Algorithm 8. In the algorithm, we have used a geometric annealing schedule. Hence, besides and , the algorithm accepts the coefficient for this schedule and, based on the experiment results, was used.
The N-Switch Reduction strategy bases on established N-Switch Coverage concept used in FSM testing[29, 26, 1] and is defined by Algorithm 9. The algorithm accepts the same inputs as the PSMT strategy (see Algorithm 1) and produces as an output. The algorithm first generates all paths in of length , where , which are put to the initial . Then, the algorithm reduces to keep only paths that starts in and ends in some vertex from . Then, paths of are further analyzed if more paths start in a particular vertex from and if so, only one of these paths is kept in .
All presented algorithms are implemented in the developer version of the Oxygen111http://still.felk.cvut.cz/oxygen/ platform [9, 7]. The platform allows for the creation of the SUT model in its graphical editor. It also allows to import and export of the SUT models in open XML-based formats. The selected algorithms can generate test path sets for a set of SUT models. Test paths can then visualized in the model and exported. An example of Oxygen user interface with visualized test path in a SUT model is given in Figure 1. SUT model states from , and are highlighted by green, red and orange color, respectively. The developer version of the platform allows for batch execution of the present algorithms on multiple SUT models and export of summary results for further analysis and evaluation.
V-a Experiment set-up and method
For the experiments, we used 180 problem instances . These instances are composed of: (1) state machine models of real industrial project, (2) modifications of the industrial real models, and (3) problem instances generated artificially. Regarding the industrial FSMs, six models describing various parts of tested Skoda cars were used. The Skoda Auto testing team created these models in a special Skoda version of the Oxygen tool supporting . These models were further modified by adding cycles to an FSM, adding possible test starts and test ends, adding or removing a state, and adding and removing a transition. In this paper, by a cycle, we mean a configuration of edges that allow non-empty trails in which only the first and last vertices are equal. The result was 24 problem instances. Another 12 problem instances were created by the same method for FSMs for data transmission protocols and mission management in the Digital Triage Assistant project222https://www.natomultimedia.tv/app/asset/656263, a joint project of CTU in Prague, Johns Hopkins University, University of Defence, NATO ACT IH and other partners. Non-disclosure agreements and confidentiality restrictions of both projects allow making publicly available only abstracted topology of FSMs without the names of states and transitions.
To create enough variety of possible FSMs, we generated an additional set of 144 problem instances by a specialized tool, developed as a master thesis by Richard Sadlon333https://dspace.cvut.cz/handle/10467/97079, CTU in Prague. The tool generates problem instances by expected properties of the graph entered as an input. In this process, , , number of cycles, , , , and can be specified.
Table I presents the selected properties of all problem instances used in the experiments. In Table I, denotes number of cycles, denotes average length of these cycles, denotes number of groups of parallel edges present in (in these groups, edges start and end in the same vertex), denotes total number of parallel edges in , denotes average node incoming degree, denotes average node outgoing degree, and denotes average node degree. Further, denotes the state in which a test path can both start and end.
Regarding the priorities defined in the SUT model, as well as were set to range 0 to 3 for all , resp. , for all problem instances. Then, was set to 2 and was set to 3 in a unified way for the experiment. In Table I, denotes number of for which and I, denotes number of for which .
To evaluate the effectiveness of the generated to detect defects, the problem instances were extended by fictional defects of two types. Type 1 Defect is present at an edge representing FSM transition and is considered to be activated when a visits . In Table I, number of these defects in problem instances is denoted as .
Type 2 defect is present at two edges and , where there is a path in from to . To consider Type 2 defect to be activated, a visits and then visits . Type 2 Defects simulate data consistency defects, when a transition induces an internal inconsistency to a SUT and other transition () causes a defective behavior of the SUT.
In Table I, the number of Type 2 defects in problem instances is denoted as and average distance between and in number of edges is denoted as .
We evaluated the following properties of : , total length of all in number of edges (denoted as ), average length of all in number of edges (denoted as ), number of unique edges in all (denoted as ). We also analyzed that expresses how many non-unique FSM transitions ( edges) need to repeat in a test path to test the unique transitions. Higher indicates higher ”edge duplication” in .
To evaluate the defect detection potential of , we analyzed the number of simulated defects of Type 1 and Type 2 that were activated by a , denoted as and , respectively. The last property to evaluate is averaged number of simulated defects activated by one test path step, defined as and for Type 1 and Type 2 simulated defects, respectively.
We ran all presented algorithms for both PSMT Basic Coverage and PSMT Extended Coverage criteria. In each of these criteria, we present and discuss the results for two test path length ranges: and . If no was returned for a particular problem instance for the individual configuration of test path length range and the coverage criterion by any of the compared algorithms, the result record was not taken into account for all algorithms for this problem instance. Because of the non-deterministic nature of the Genetic Algorithm and Simulated Annealing test path reduction variants of the PSMT strategy, we ran all computations three times and averaged the results.
Table II presents the averaged results for all problem instances for all compared algorithms and both coverage criteria for the test path length range . The major properties we further analyze and discuss are , and