An Approximation-based Approach for the Random Exploration of Large Models

by   Julien Bernard, et al.

System modeling is a classical approach to ensure their reliability since it is suitable both for a formal verification and for software testing techniques. In the context of model-based testing an approach combining random testing and coverage based testing has been recently introduced [9]. However, this approach is not tractable on quite large models. In this paper we show how to use statistical approximations to make the approach work on larger models. Experimental results, on models of communicating protocols, are provided; they are very promising, both for the computation time and for the quality of the generated test suites.



page 1

page 2

page 3

page 4


DeepEvolution: A Search-Based Testing Approach for Deep Neural Networks

The increasing inclusion of Deep Learning (DL) models in safety-critical...

Model-based Automated Testing of JavaScript Web Applications via Longer Test Sequences

JavaScript has become one of the most widely used languages for Web deve...

Hybrid Intelligent Testing in Simulation-Based Verification

Efficient and effective testing for simulation-based hardware verificati...

Model-based Automated Testing of Mobile Applications: An Industrial Case Study

Automatic testing of mobile applications has been a well-researched area...

DRLGENCERT: Deep Learning-based Automated Testing of Certificate Verification in SSL/TLS Implementations

The Secure Sockets Layer (SSL) and Transport Layer Security (TLS) protoc...

Enhancing Path-Oriented Test Data Generation Using Adaptive Random Testing Techniques

In this paper, we have developed an approach to generate test data for p...

Complete Agent-driven Model-based System Testing for Autonomous Systems

In this position paper, a novel approach to testing complex autonomous t...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Many critical tasks are now assigned to automatic systems. In this context, producing trusted software is a challenging problem and a central issue in software engineering. Recent decades have witnessed the strengthening of many formal approaches to ensure software reliability, from verification (model-checking, automatic theorem proving, static analysis) to testing, which remains an inescapable step to ensure software quality. A great effort has been made by the scientific community in order to upgrade hand-made testing techniques to scalable and proven framework.

Experience shows that random testing is a very efficient technique for detecting bugs, especially at the first stages of testing activities. The strength of random testing consists of its independence on tester’s priority and choices. However, the nature of random testing is to draw randomly a test rather than choosing it

, and it is therefore inefficient to detect behaviour of a program occurring with a very low probability. In 

[DBLP:journals/sttt/DeniseGGLOP12], a random testing approach consisting of the exploration of large graph based models has been proposed. In order to tackle the problem of low probabilistic behaviour, the authors have also suggested to bias the random generation, by combining it with a coverage criterion, in order to optimize the probability to meet system’ features described by this criterion. It however requires the computation of large linear systems, which becomes rapidly intractable in practice for large graphs.

In this paper we propose a sampling-based approach in order to compute approximated values of the system’ solutions, deeply improving the efficiency of the computation. Experimental results on various graphs provided in the paper show a very significant time computation improvement while keeping similar covering statistical properties.

1.1 Related Work

A prevailing methods in model-based testing consists in designing the system under test by a graph-based formal model [DBLP:journals/stvr/Wah96, lee1996principles] on which different algorithms may be used to generate the test suites. This approach has been used for a large class of applications from security of Android systems [DBLP:conf/icst/TangCZGXHBM17] to digital ecosystems [DBLP:conf/icst/LimaF16]. A large variety of models can be used for model-based testing such as Petri nets [DBLP:conf/icst/ThummalaO16], timed automata [DBLP:conf/icst/WangPB17], pushdown automata [DBLP:journals/stvr/DreyfusHKM14], process algebra [DBLP:journals/infsof/AlbertoCGS17], etc. Moreover, a strength of model-based testing is that it can be combined with several verification approaches, such as model-checking [DBLP:journals/stvr/DadeauHKMR15] or those using SMT-solvers [DBLP:conf/tap/AichernigJK13]. A general taxonomy with many references on model-based testing approach can be found in [DBLP:journals/stvr/UttingPL12].

Random testing approaches have been introduced in [DBLP:conf/icse/DuranN81] and are widely used in the literature, either for generating data [DBLP:conf/pldi/GodefroidKS05, DBLP:conf/icst/HeamN11] or for generating test suites [DBLP:conf/soqua/Oriat05]. As far as we know, the first work combining random testing and model-based testing has been proposed in [DBLP:conf/issta/GroceJ08] as a combination of model-checking and testing. In [DBLP:journals/sttt/DeniseGGLOP12] the authors have proposed an improved approach to explore the models at random. This technique has been extended to pushdown models [DBLP:conf/tap/HeamM11, DBLP:journals/stvr/DreyfusHKM14] and to grammar-based systems [DBLP:conf/icst/DreyfusHK13].

1.2 Formal Background

Figure 1: Illustrating example.

For a general reference on probability theory, see 


Finite Automata.

Models considered in this article are finite automata, that are labelled graphs. More precisely, a finite automaton is a tuple , where is a finite set of states, is a finite alphabet, is the set of transitions, is the set of initial states, and is the set of final states. A path in a finite automaton is a sequence of transitions. The integer is the length of the path. If and in , is said successful. The path visits a state if there exists such that . An automaton is trim if every state is visited by at least one successful path. All automata considered throughout this paper are trim. An example of an automaton is depicted in Fig. 1 : its set of states is , the alphabet is , its set of transitions is

its set of initial states is reduced to and all its states are final.

Let be a -state automaton and . We denote by the automaton on the alphabet whose set of states is (two copies of ) and :

  • Its set of initial states is ,

  • Its set of final states is ,

  • Its set of transitions is .

Intuitively, a successful path in starts with an initial state of the form and remains in a state of the form until it visits . Then, if is final in it may ends or continue with states of the form . One can easily show that there is a bijection between the set of successful paths of of length and the set of successful paths of of length visiting . denoted Let us consider for instance the automaton depicted in Fig. 1 and state 3. The corresponding automaton is depicted in Fig. 2 (states (4,0), (1,1) and (2,0) have to be removed to make the resulting automaton trim).

Figure 2: Illustrating example for constrained paths.

Automata used in many testing applications have a bounded outgoing degree. Throughout this paper, we consider that . Note that it is not a theoretical requirement: it is only used for the complexity issues. Indeed, all proposed algorithms work for any automata. Under this hypothesis, computing can be done in time and the resulting automaton has at most twice the number of states (regarless of the fact that ).

Counting Paths.

We call NumPaths an algorithm that, given a finite automaton and a positive integer , computes the number of successful paths of length in . We call RandomPath an algorithm that, given a finite automaton and positive integers , randomly, uniformly and independently generates successful paths of length in . Several algorithms have been developed for processing NumPaths and RandomPath [DBLP:journals/tcs/OudinetDG13]

, whose complexities depend on several parameters. Let us observe, without going into details, using floating point arithmetics, that

NumPaths can be performed in , where is the number of states of . And RandomPath can be performed in time . Note that the different approaches may have different meanings of time/space complexities, both for the preprocessing step and the generation step. The reader can see [DBLP:journals/tcs/OudinetDG13, Table 4] and [DBLP:journals/algorithmica/BernardiG12, Table 1] for more details.

Random Biased Exploration of Finite Automata.

The objective is here to biased the random generation of paths (i.e. not use a uniform random generation) in order to improve the state coverage of the automata. It is necessary to provide a quite detailed description of the algorithms in [DBLP:journals/tcs/OudinetDG13]. The first approach, denoted later Uniform, consists in uniformly picking up a given number of paths from the set of successful paths of a given length. The approach can be applied to very large graphs with hundreds of nodes (see [DBLP:journals/sttt/DeniseGGLOP12, Section 6]). However, rare events can be missed up, and in order to optimize111Computing test suites of a reduced size is a major issue in the testing process, since executing test on the system is frequently a complex issue (not adressed in this paper). the coverage criterion (let us present it here for nodes coverage222The approach can easily be adapted for transitions coverage.) of the graph, the following approach, denoted later Exact, is proposed to produce successful paths of an automaton whose set of states is :

  1. Choose a set of successful paths (for instance those of length less than or equal to a constant ),

  2. For each pair of nodes, compute the probability that a path of visiting also visits ,

  3. Solve the linear programming system whose variables are



    Solution is a distribution of probabilities over the states of the automaton,

  4. Repeat times: pick a node up at random according to the distribution . Pick up at random (uniformly) a path visiting .

The goal of the linear programming system is to optimize the minimal probability of a state to be visited by a random path.

Let us illustrate this approach on the example depicted in Fig. 1. Note that if the goal is to cover a given proportion of the set of states (for instance) Step 4. can be replaced by: generate paths as soon as the wanted proportion of states are visited by these paths. There are successful paths of strictly positive length less than or equal to  reported in Table 1. Since the automaton is deterministic, one can identify successful paths with their labels. Let be this set of paths.

length paths number of paths
1 , 2
2 , , , 4
3 , , , , , , , , , 10
Table 1: Successful paths of length less than or equal to 3 for Example 1.

There are out of paths of visiting state . Therefore, the probability of visiting state  by uniformly generated paths of is . In order to generate a path visiting , one has to generate averagely tests. Moreover, for this example ’s matrix is

For instance, for every since all paths visit . Similarly, since all paths visiting also visits . There are four paths (, , and ) of needed length visiting and, among these paths, only visits . Therefore . The resolution333Resolutions have been performed using the lp_solve solver. of linear programming systems (1) provides in this context the solution: , , , and . In this context, the biased approach covers all states averagely with less than generated paths.

The bottleneck of this approach is Step 2. since computing the s requires many manipulations on the graphs (it requires to compute the ): for each , Algorithm NumPaths has to be applied to graphs 4 times larger than the initial ones. The complexity is in with quite large involved constants, making the approach intractable for big ’s.

1.3 Contributions

In this paper we propose to not exactly compute the s but instead to approximate them by using statistical sampling, as described in Section 2. Experimental results on several examples of communication protocols models are provided in Section 3. The paper reports on very promising experimental results: the computation time is significantly better for a similar quality of the large graphs coverage.

2 Approximating the Linear Programming Systems

In this section, we propose to approximate the coefficients by by using classical sampling techniques. Using times Algorithm RandomPath, one can count as the number of paths visiting , and the number of paths visiting both and . If then .

2.1 Approximation Algorithm

More precisely, let there be a trim finite automaton , a strictly positive integer , a strictly positive integer and a strictly positive integer  (the parameter

is used to provide some bounds on the precision of the approximation: each evaluation of a parameter is estimated using a sample of size at least 


  • (Step 1): Generate successful paths in of length less than or equal to uniformly. For each , let be the number of these paths visiting , and be the number of these paths visiting both and .

  • (Step 2): For each ,

    • If and , then let ,

    • If , let ,

    • If , generate paths visiting and set as the proportion of these paths visiting .

  • (Step 2): For each , .

Let us illustrate the approach on the example depicted in Fig. 1, with and . Rather than compute exactly the ’s, we randomly and uniformly generate 1000 paths of length less than or equal to 3. We obtain the following matrix for the  :

The resolution of systems provides the solution , , and .

In this example, there are 243 paths visiting , 826 paths visiting and 288 paths visiting . Therefore, running the algorithm with will change the second column of the matrix since . In this case, the automaton for the paths visiting is computed. Generating paths visiting state  provides the following matrix:

The resolution of systems provides the solution , , and .

Section 3 describes more experiments and provides details, both on the quality of the results and on the time to compute the ’s.

Notice too that, as mentioned in [DBLP:journals/sttt/DeniseGGLOP12], the optimal solution leads to a loss of randomness: many ’s a null. It is proposed in [DBLP:journals/sttt/DeniseGGLOP12] to fix minimal probability to the ’s. It can be directly adapted in our approach by adding, in the programming linear system, some inequations of the form . This situation would not be considered in the experiments developed in this paper.

2.2 Complexity

We investigate in this section the worst case complexity of the proposed algorithm. Step (1) can be performed in time : first the paths are generated in time . These paths are not stored but a table of size is filled in the following way: if the -th path visits state , and otherwise. It is done on the fly and in time . The are calculated by computing columns sums in time too. Similarly, each can be computed in time . Therefore, computing all of them is performed in time .

Step 2-(a) is performed in time as well as Step 2-(b). Step 2-(c) is performed in time : computing the specific automaton is done in time (under the hypothesis that the number of transitions is in ).

Step 3 is performed in time .

In conclusion, if we denote by the number of calls to Step 2-(c), the complexity is: .

A small (for instance ) will provide a small (), but a coarser approximation, as exposed in the next section.

2.3 Precision of the ’s

Each is the parameter of Bernoulli’s Law (see [DBLP:books/daglib/0012859, Section 2.2]). The precision of the estimation can classically be obtained using either Bienaymé-Chebyshev’s Inequality [bienayme, cheb] or Hoeffding’s Inequality [hoeff].

First, assuming that , then Bienaymé-Chebyshev’s Inequality provides for any :

and if , it provides:

In order to have an precision with a 0.95 confidence level, has to be fixed to (this is an upper bound).

Secondly, one can have another evaluation using Hoeffding’s Inequality (better in most of cases): for any ,

In order to have an precision with a 0.95 confidence level, has to be fixed to (this is also an upper bound).

Let us note that the two above inequalities provide upper bounds that are not very tight: for states frequently visited by random paths, will be significantly greater than , and the estimation of the algorithm will be very precise. As it is shown in the next section, running the algorithm with frequently provides very acceptable solutions and very good solutions with . For the two bounds above do not ensure precise estimations: Hoeffding’s Inequality states that with a 0.8 confidence level we have an estimation of with . An hypothesis explaining why works is that it is important to detect whether while visiting the probability to also visit is significant. But it’s not critical to know how significtant it is, for instance if or ; it is important to know that generating a path visiting will quite frequently provide a path visiting .

Finally, other statistical tools can be used to obtain bounds on

, for instance the well-known central limit theorem.

3 Experiments

RW 4.38 2–10 4.38 2–10 4.38 2–10 4.38 2–10
Uniform 4.22 2–11 4.22 2–11 4.22 2–11 4.22 2–11
Approx 10 4.65 2–13 4.65 2–13 4.65 2–13 4.65 2–13
Approx 1000 4.18 2–12 4.18 2–12 4.18 2–12 4.18 2–12
Exact 4.42 2–12 4.42 2–12 4.42 2–12 4.42 2–12
15 states
18 transitions

RW 428 18–1724 790 159 – 2062 2451.4 599–7727 2451.4 599–727
Uniform 22.58 9–45 35.45 14–79 92.36 32–216 92.36 32–216
Approx 10 11.7 7–21 17.2 10–52 39.3 16-181 39.35 16 – 181
Approx 1000 10.46 7–20 14.9 8.31 30.9 11–73 30.9 11–73
Exact 11.23 7–24 15.5 8–30 29.8 13–71 29.8 13–71

Dekker1, 86 states, 178 transitions

RW 15 9-27 21.5 13–37 37.8 18–70 50.5 19–130
Uniform 31.2 16–51 47.7 22–88 87.6 40–166 115.5 45–278
Approx 10 19.5 13–34 28.1 17-51 48.6 27–146 67.9 30–146
Approx 1000 18 10–29 24.5 13-38 40.3 20–74 50.0 26-86
Exact 18.4 12–30 24.8 18–38 39.5 23–63 49.6 26–102
Fsm1, 120 states, 582 transitions

RW 102.1 8-430 178.3 16–735 330.6 16–1125 330.6 16–1125
Uniform 9.7 2–35 13.1 2–39 21.9 2–90 21.9 2–90
Approx 10 2.9 2–11 3.5 2–11 5.1 2–23 5.1 2–23
Approx 1000 3.0 2–8 3.0 2–8 3.5 2–8 3.5 2–8
Exact 3.3 2–9 3.3 2–9 3.7 2–9 3.7 2–9

Moesi2, 22 states, 43 transitions

RW 11.5 6–18 80.6 45–153 122.8 73–247 260.8 120–514 342.4 156–649
Uniform 8.78 7–13 54.4 36–82 80.9 51–12 180.972–333 277.3 105–776
Approx 10 7.49 5–10 37.6 27–56 53.3 38-91 102.6 56–205 140.6 59–347
Approx 1000 7.6 6 – 11 35.8 24–51 50.9 32–80 88.11 58-147 106.8 62–179
Exact 7.6 5–10 34.9 24–46 47.6 33–63 82.3 55–121 101.5 56–165

Kanban1, 160 states, 1151 transitions

Figure 3: Comparative results (1) for number of generated tests
RW 54991.6 39414–69917 155803 117044-214680
Uniform 937.1 827–1044 1493 1240–176
Approx 10 789..9 718–884 1129.7 980–1292
Approx 1000 704.7 651–759 974.1 892–1066
Exact 708 655–769 698.8 867–1101
Uniform 3405.7 2760–4186 11049 6563–22480
Approx 10 1998.8 1707–2282 11962 3336–62763
Approx 1000 1625.3 1441–1841 3410.8 2265–7144
Exact 1610.5 1402–1843 3268.8 2369–4738
Ttp8, 3201 states,
6765 transitions

RW 505 248–850 1073 460 – 1777 5702.6 1395–16035
Uniform 67.6 47–92 106.6 73–160 226.6 115-471
Approx 10 53.05 40–73 73.3 55–106 134.1 90–226
Approx 1000 49.1 39–64 67.6 51.95 115.1 79–199
Exact 49.7 39–72 66.8 51–95 114.4 76–168
RW 21750 2579 – 119811
Uniform 372.3 165–809
Approx 10 193.3 97–439
Approx 1000 158.1 85–300
Exact 157.4 91–270

Prodcons10, 286 states, 600 transitions

Figure 4: Comparative results (2) for number of generated tests
Name States Transitions Eccentricity Nb. of paths
Barber1 15 18 5 74
Berkeley3 1376 3974 51
Consistency3 806 1206 600
Csm1 24 57 8
Dekker1 86 178 17
Dragon3 103 696 50
Fms1 120 582 14
Illinois3 103 307 100
Kanban1 160 1151 14
Lift3 499 587 302
Moesi2 22 43 11
Prodcons10 286 660 20
Ttp8 3201 6765 32
Table 2: Graphs used for benchmarking.

This section is dedicated to an experimental evaluation of the proposed approximation-based approach. In Section 3.1 the set of used automata is described. Section 3.2 explains the experimental protocol. Finally, the obtained experimental results are provided in Section 3.3, both for the quality of the approach and for computation time.

3.1 Benchmark

Experiments have been done on several automata modeling communication protocols designed for the FAST tool [DBLP:conf/cav/BardinLP06] available444 online as a library of parametric counter automata (the parameter can be, for instance, the number of communicating processes). For several examples and parameters, the counter automaton has been faltered into a classical finite automaton. The list is given in Table 2: first column contains the name of the protocol with the value of the parameter. The second and the third columns respectively report on the number of states of the automaton and the number of transitions. The fourth column provides the eccentricity555Eccentricity is an important parameter since it is the minimal length required for paths to have a chance to visit each state. of the automaton, that is the maximal distance of an edge to the initial states. Finally, the last column gives the approximate number of successful paths in the automaton of length less than or equal to twice the eccentricity. Note that in these graphs all states are final.

3.2 Experimental Protocol

For each protocol, we have measured the number of tests/generated paths required to cover either , or , or , or , or of the states. Several values close to have been chosen since many biased approaches have been introduced to handle rare events, and many methods will efficiently cover or of the graph. It is harder to cover the remaining last states. We have compared 5 different approaches. First, the RW Approach consists in performing isotropic random walks in the automaton: once in a state, the next one is picked up uniformly among its neighbours. The path ends either when it reaches a dead-end state, or when its length is twice the eccentricity. The second approach, denoted Uniform, is the one introduced in [DBLP:journals/sttt/DeniseGGLOP12]: paths of length bounded by twice the eccentricity are uniformly generated. The approach denoted Exact is the biased approach proposed in [DBLP:journals/sttt/DeniseGGLOP12], where the linear system is exactly computed. The Approx 10 and Approx 1000 approaches are the ones proposed in this article: for [resp. ] the ’s are approximated using [res. ] randomly generated paths, where is the number of states.

Note that comparing the distribution given by the exact approach and the approximation-based approaches is not easy. Indeed, a linear programming system may have different optimal solutions. Let us consider for instance the example depicted in Fig. 5. The set of successful paths visiting is the same as the set of successfully paths visiting . Therefore, in any optimal solutions of the linear programming system given and , one can do the following changes: and with , and we also obtain an optimal solution.

Figure 5: Illustrating example with different optimal solutions.

3.3 Qualitative Experimental Results

Since the test generation procedures are randomized, performance is stochastic. For each example and each coverage proportion, each approach has been experimented 100 times. For each case, we report on the average number of tests obtained in order to cover the wanted proportion, but also the minimum number of tests (the best case), and the maximum number of tests (the worst case).

Results presented in Figs. 3 and 4 are obtained with  : there is no a priori guarantee on the precision of the approximations. Results presented in Fig. 6 are obtained with and with (and in one case with ). For instance, the second table in Fig. 3 reports on the result for Dekker1: in order to cover of the set of the states, the RW approach requires on average 790 paths. In the best case (of the experiments), it only requires 159 paths, and in the worst case 2062 paths have been generated. For the same coverage, the Uniform approach requires 35.45 paths in average. The Exact approach only requires 15.5 paths in average.

Relatively to the other approaches, the performance of RW deeply depends on the topology of the automaton. For instance, for Prodcons10 or Ttp8 or Moesi2, RW is ugly, and requires much more tests to (partially) cover the set of states. For Fms1, RW is as efficient as Exact. For some automata, there is no result for RW: after hours of computation, the approach was not able to cover 90% of the set of states. In these cases, some states occur with a so low probability on random walks, that in practice it is not possible to generate a path visiting them.

One can see that for Barber1, Dekker1, Fms1, Moesi2, Kanban1, Ttp8 and Prodcons10, all biased approaches are better (cover the set of states with less paths) than the uniform one. Moreover, the Exact approach is better than the approximate ones, but not significantly with the Approx 1000. Consider for instance Fms1: the Uniform approach requires on average 87.6 paths to cover 99% of the states. With Approx 10 this number falls to 48.1, and it falls to 40.3 with Approx 1000. The Exact approach requires 29.8 paths on average.

The results for Lift3 are similar but the Approx 1000 is not so close to the Exact approach. For Berkeley3, Illinois3 and Dragon3, the Exact approach is clearly more efficient to cover the set of states. It is similar for Consistency3, but only for the 100% coverage criterion. A significant case is Illinois3: the Exact approach requires on average a unique path to cover all states, while the Approx 1000 approach requires 47 paths. For all these examples there is a huge number of paths, and many states are visited with a very low probability by a path: the corresponding ’s are set to zero since , thus providing a very bad approximation. For instance, for Illinois3, 84 states over the 103 states are not visited by any random paths. We run the experiment with Approx 1000 and . The obtained results are presented in Fig. 6: these results are much better and close to the ones of the Exact approach.

In conclusion, for the quality of the coverage, running Approx 10 with seems to be an efficient solution.

Uniform 176 153–240 315 258–375 630 486–834 1203 837–2376
Approx 10 166 142–212 301 253–377 648 431–884 2127 797–5611
Approx 1000 171 259–381 307 259–381 673 485–1070 1973 807 – 4485
Exact 166 143 – 203 294 256–346 589 440–816 1084 689–1783
Approx 10 168 140–231 297 256–376 626 479–780 1474.9 730 – 4749
Approx 1000 168 133–211 300 258–359 624 489–828 1390 765 – 4042
Approx 10 168 141–208 300 250–373 610 480–848 1292 710 – 2715

Consistency3, 806 states, 1206 transitions

Uniform 10.56 4–30 23 6–100 64.7 10–400 85.3 10–400
Approx 10 5.4 3–14 9 4–46 24.7 8–107 34.9 9–137
Approx 1000 4.4 3–7 6.6 4–23 13.9 6-37 18.9 7-51
Exact 3.9 3–6 5.5 4-9 9.5 6–20 13.1 6-24
Approx 10 3.8 3–4 5.4 4–7 8.9 6–15 12.9 8–23
Approx 1000 3.7 3–5 5.3 4–7 9.0 7–14 12.4 7-20

Lift3, 499 states, 587 transitions

Uniform 22.5 7–101 35.8 12–135 69.6 24–267 112.1 34–317
Approx 10 18.3 9–47 29.1 12–76 63.2 15–231 98.5 34–267
Approx 1000 14.7 6–29 25.2 11–55 56.7 13–184 87.2 13–319
Exact 10.8 6–17 15 7–25 22.8 10–41 29.9 10–57
Approx 10 () 10.6 5–17 14.6 6–29 23.5 12–47 30.7 16–79
Approx 1000 () 10.1 6–18 14.3 8–26 22.6 14–42 29.5 14–79
Dragon3, 103 states, 696 transitions

Uniform 337 235–448 594 400-863 1551 993–2262 4470 2316–9974
Approx 10 222 167–286 384.8 258-536 1014.6 655-1382 2886.6 1504–5251
Approx 1000 189 134–247 323 232–498 854 467–1302 2116 1108–2886
Exact 103 60–185 137.0 84–210 199.6 109–364 224.1 128–510
Approx 10 (r=10) 102 66-149 138 80-280 207.0 107–399 233.6 140–412
Approx 1000 (r=10) 102 67–202 137 86-256 206 112–423 231 122–508

Berkley3, 1376 states, 3974 transitions

Uniform 8.8 1–46 16 1-73 40.1 1–209 68.43 1–338
Approx 10 7.5 1-27 12.1 1–46 30.1 1–150 56.9 1–314
Approx 1000 6.2 1–38 11.25 1-61 29.1 1-210 48.3 1–234
Exact 1.0 1-2 1.0 1-2 1.0 1-2 1.1 1-2
Approx 10 1.0 1–2 1.0 1–2 1.1 1–2 1.2 1–3
Approx 1000 1.0 1–2 1.0 1–2 1.1 1–2 1.2 1–2
Illinois3, 1524 states, 307 transitions
Figure 6: Comparative results (3)

3.4 Computation Time

(seconds) Berkeley3 Consistency3 Dragon3 Lift3 Illinois3
Exact 26401 40964 29 4337 18
Approx 10 () 1 58 1 8 0.4
Approx 1000 () 186 5794 21 813 39
Approx 10 () 16 110 1 12 1
Approx 1000 () 208 5890 25 862 41
Approx 10 () 124 -
Table 3: Time to compute the linear programming system.

Let us note first that for all approaches, generating paths is done practically in a very efficient way. As mentioned before, the bottleneck step is the computation of the linear programming system. In Table 3, the time (in seconds) used to compute the linear programming system is given for the protocols Berkeley3, Consistency3, Dragon3, Lift3 and Illinois3. The results are similar for the other protocols. For Illinois3, using Approx 1000 is less efficient than using the Exact approach. The reason is that the automaton is quite small. However, for other cases, using the approximation-based approaches is faster. And it is significantly faster for large automata. For instance, for Consistency3, while the Exact approach requires more than 11 hours, and only about 90 minutes are needed for Approx 1000 (with ).

In all cases, the best compromise seems to use Approx 10 with : the computation time is strongly better, and the quality of the biased approach is similar to the Exact approach, except for Consistency. For this protocol, we run the Approx 10 with and we obtain better results, closer to the Exact approach, with a very short computation time (about 2 minutes, in comparison to 11 hours for the Exact approach).

3.5 Experiments on Large Graphs

We have experimented the approaches on a model of the Centralserver2 protocol, which has 2523 states and 18350 transitions, an eccentricity of 63, and about successful paths of length less or equal to 126. By computing the first ’s, we estimate that the computation time of the linear programming system with Exact will require about 200 days. The linear programming system with Approx 10 and Approx 1000 () has been computed respectively in 81 seconds and in 24 minutes. The obtained qualitative results compared to Uniform are given in Table 4.

Uniform 680 573–786 1198 947–1492 2942 2358–3612 9413 5487–19533
Approx 10 () 316 287–349 476 437–529 878 775–1037 2065 1223–4337
Approx 1000 () 313 281–345 467 415–515 864 729–1037 1926 1252–3514
Table 4: Results for Centralserver2.

We also used the algorithm proposed in [DBLP:conf/stacs/CarayolN12] to randomly generate two trim automata with respectively 5659 states (with 17007 transitions) and 11251 states (with 33753 transitions). The approximated linear programming system obtained by the A͡pprox 10 and Approx 1000 approaches (with ) has been computed in respectively 5.5s and 613s for the first graph, and in respectively 26.3s and 1162s for the second graph.

4 Conclusion

In this paper we proposed an approximation-based approach for the random biased exploration of large models. It has been experimented on several examples: in practice the approximation is not too coarse, and the quality of the generated test suites to cover the states of the model is excellent compared to the exact approach and to the other random approaches. For computation time, using approximation is significantly better since the approach can be used on graphs with more than 10000 states. In the future we plan to investigate recent advances in optimization in order to improve the computation time.