Fault Detection for Timed FSM with Timeouts by Constraint Solving

by   Omer Nguena Timo, et al.

Recently, an efficient constraint solving-based approach has been developed to detect logical faults in systems specified with classical finite state machines (FSMs). The approach is unsuitable to detect violations of time constraints. In this paper, we lift the approach to generated tests detecting both logical faults and violations of time constraints in systems specified with timed FSMs with timeouts (TFSMs-T). We propose a method to verify whether a given test suite is complete, i.e., it detects all the faulty implementations in a fault-domain and a method to generate a complete test suite. We conduct experiments to evaluate the scalability of the proposed methods.



page 1

page 2

page 3

page 4


Mixed-level identification of fault redundancy in microprocessors

A new high-level implementation independent functional fault model for c...

Metamorphic Relation Prioritization for Effective Regression Testing

Metamorphic testing (MT) is widely used for testing programs that face t...

Complete Requirements-based Testing with Finite State Machines

In this paper, new contributions to requirements-based testing with dete...

DeepMutation: A Neural Mutation Tool

Mutation testing can be used to assess the fault-detection capabilities ...

Automatic Property-based Testing of GraphQL APIs

In recent years, GraphQL has become a popular way to expose web APIs. Wi...

Automatic Generation of RAMS Analyses from Model-based Functional Descriptions using UML State Machines

In today's industrial practice, safety, reliability or availability arti...

Wikidata Constraints on MARS (Extended Technical Report)

Wikidata constraints, albeit useful, are represented and processed in an...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

The fault domain coverage criterion can be adopted to generate tests revealing faults in safety/security critical systems under test (SUT) [2]. The domain can be built from referenced databases111E.g.: https://nvd.nist.gov/ or expert knowledge. Efficient test generation methods are needed especially for the fault domains of important sizes, which has motivated the development of an approach [18, 19] leveraging on recent advances in the field of (Boolean) constraint solving. The approach has been elaborated to detect logical faults in reactive systems specified with finite state machines (FSMs). We plan to lift the approach to detect both logical faults and violations of time constraints in reactive systems; especially, we focus on reactive systems specified with timed FSMs with timeouts (TFSMs-T).

TFSM-T [15, 4] is an extension of the classical FSM with timeout transitions for expressing time constraints. They define timeouts and the next states to be reached if no input is applied before the timeouts expire; otherwise outputs defined by input/output transitions are produced. Although they express limited types of time constraints as compared to other timed FSMs [4], TFSMs-T have been used to specify reactive systems such as web applications [27] and protocols [26, 22, 13]. Logical faults in TFSMs-T correspond to unexpected outputs or unexpected state changes. Reducing and increasing waiting time are violations of time constraints. An implementation under test for a given specification TFSM-T can be represented with a mutated version of the specification TFSM-T also called a mutant. A mutant can be obtained by seeding the specification with an arbitrary number of faults. A fault domain for a specification is then a finite set of possible mutants; it can be built from a list of identified faults to be detected in systems under test. A mutant is nonconforming if its timed output sequence differs from that of the specification for some timed input sequences (tests). A complete test suite for a fault domain detects all nonconforming mutants in the domain.

Model-based testing with guaranteed fault coverage has been investigated for untimed and timed models. Finite state machines can be preferred over label transition systems for representing systems. This is probably because FSMs have been used early in testing digital circuits 

[25] and protocols [3], and they do not permit nondeterministic choices between the application of inputs and the production of outputs. Several approaches have been investigated for FSM-based test generation with guaranteed fault coverage [3, 25]. FSMs have been extended to express time constraints, which has resulted in a variety of timed FSMs  [16, 4, 9, 27, 10]. Timed FSMs are not compared to the well-known timed automata [1] for which testing approaches have been developed [14, 8, 6] . Testing approaches [9, 27, 10, 24] for timed FSMs integrate the reasoning on time constraints in well-known FSM-based testing approaches [25, 5]. The work  [12]

evaluate the application of different meta-heuristic algorithms to detect mutants of Simulink models. Meta-heuristic algorithms do not guarantee the detection of all pre-defined nonconforming mutants. The methods in  

[18, 19] to verify and generate a complete test suite for FSM specifications are based solving constraints or Boolean expressions, which allows to take advantage of the efficiency of constraint/SAT solvers [7, 23]; this is a novelty as compared to the work in [21] and the well-known approaches such as the W-method. The high efficiency of using constraint solving in testing software code was demonstrated in  [11]. The constraints specify the mutants surviving given tests; they are defined over the transitions in executions of the mutants. The executions are selected with a so-called distinguishing automaton of the specification and the fault domain that is compactly modeled with a nondeterministic FSM called a mutation machine. A solution of the constraints is a mutant which, if it is nonconforming, allows to generate a test detecting the mutant and many others; then the constraints are upgraded to generate new tests.

Our contribution is to lift the methods in  [18, 19] for verifying and generating complete test suites for fault domains for TFSM-T specifications. In our work, specifications and mutants are deterministic and input-complete TFSMs-T. We define a new distinguishing automaton with timeouts for a TFSM-T specification and a fault domain. The automaton serves to extract transitions in detected mutants and build constraints for specifying test-surviving mutants. Extracting the transitions, we pair input/output transitions with timeout-unexpired transitions allowing to pass the input/output transitions; this is formalized with a notion of "comb". We have implemented the methods in a prototype tool which we use to evaluate the efficiency of the methods and compare our results with those of the related work.

Organization of the paper.

The next section introduces a fault model for TFSMs-T and the coverage of fault models with complete test suites. In Section 3 we build constraints for the analysis of timed input sequences and the generation of complete test suites. The analysis and generation methods are presented in Section 4. Section 5 presents an empirical evaluation of the efficiency of the methods with the prototype tool. We conclude the paper in Section 6.

2 Preliminaries

Let and denote the sets of non-negative real numbers and non-null natural numbers, respectively.

2.1 TFSM with timeouts

Definition 1 (TFSM with Timeouts)

A timed finite state machine with Timeouts (shortly, TFSM-T) is a 6-tuple where , and are finite non-empty set of states, inputs and outputs, respectively, is the initial state, is an input/output transition relation and is a timeout transition relation defining at least one timeout transition in every state.

Our definition of TFSM-T extends the definition in [4] by allowing multiple timeout transitions in the same state, which we use later to compactly represent sets of TFSMs-T. An input/output transition defines the output produced in its source state when input is applied. A timeout transition defines the timeout in state . A timeout transition can be taken if no input is applied at the current state before the timeout of the transition expires. It is not possible to wait for an input beyond the maximal timeout defined in the current state.

A TFSM-T uses a single clock for recording the time elapsing in the states and determining when timeouts expire. The clock is reset when the transitions are passed. A timed state of TFSM-T is a pair where is a state of and is the current value of the clock and for some such that . An execution step of in timed state corresponds either to the time elapsing or the passing of an input/output or timeout transition; it is permitted by a transition of . Formally, is an execution step if it satisfies one of the following conditions:

  • (timeout step) , and for some such that ; then is said to permit the step.

  • (time-elapsing step) , , and for some and such that there exists ; then is said to permit the step.

  • (input/output step) with , and there exists ; then is said to permit the step.

Time-elapsing steps satisfy the following time-continuity property w.r.t the same timeout transition: if is a sequence of time-elapsing steps permitted by the same timeout transition , then is a time-elapsing step permitted by . In the sequel, any time-elapsing step permitted by a timeout transition can be represented with a sequence of time-elapsing steps permitted by the same timeout transitions.

An execution of in timed state is a sequence of steps with , such that the following conditions hold:

  • ,

  • is not an input/output step,

  • is a input/output step implies that is a time-elapsing step for every .

If needed, the elapsing of zero time units can be inserted between a timeout step and an input/output step.


(a) A specification TFSM


(b) A mutation TFSM

(c) A mutant TFSM
Figure 4: Examples of TFSMs, state is initial. Dashed arrows represent mutated transitions and solid arrows represent transitions of the specification. Names of transitions appear in brackets.

Let be a non-decreasing sequence of real numbers, i.e., for every . The sequence in with is a timed input/output sequence of execution if is the maximal sequence of input/output pairs occurring in . The delay for each input/output pair , with , is the amount of the time elapsed from the beginning of to the occurrence of . The timed input sequence and the timed output sequence of are and , respectively. We let and denote the timed input and output sequences of execution . Given a timed input sequence , let denote the set of all timed output sequences which can be produced by when is applied in , i.e., is an execution of .

A TFSM-T is deterministic (DTFSM-T) if it defines at most one input-output transition for each tuple and exactly one timeout transition in each state; otherwise, it is nondeterministic. is initially connected if it has an execution from its initial state to each of its states. is complete if for each tuple it defines at least one input-output transition. Note that the set of timed input sequences defined in each state of a complete machine is .

We define distinguishability and equivalence relations between states of complete TFSMs-T. Similar notions were introduced in [27]. Intuitively, states producing different timed output sequences in response to the same timed input sequence are distinguishable. Let and be the states of two complete TFSMs-T over the same inputs and outputs. Given a timed input sequence , and are distinguishable (with distinguishing input sequence ), denoted , if the sets of timed output sequences in and differ; otherwise they are equivalent and we write , i.e., if the sets of timed output sequences coincide for all timed input sequence .

Definition 2 (Submachine)

TFSM-T is a submachine of TFSM-T if , , and .

Example 1

Figure 4 presents two initially connected TFSMs-T and . is nondeterministic; it defines two timeout transitions in states and , which is not allowed in [4]. is a complete deterministic submachine of .

Here are four executions of the TFSM-T in Figure (b)b, where the transitions defining the steps appear below the arrows:

Let us explain the first execution. It has steps represented with arrows between timed states. The label above an arrow is either a delay in or an input-output pair. The label below an arrow indicates the transitions permitting the step. The first, third and seventh steps of the first execution are time-elapsing. The second, fourth and last steps are input-output. The fifth and the sixth steps are timeout. The first step is permitted by either transition or because their timeouts are not expired units after the machine has entered state . The timeout of transition permitting the seventh step has not expired before input-output transition is performed at the last step. The difference between the first and the third execution is that input is applied lately in the third execution, i.e., time units after the third execution has started. The timed input/output sequences for the four executions are , , and , respectively. The timed input sequence and the timed output sequence for the first execution are and . Similarly, we can determine the timed input and output sequences for the three other executions. The third and the fourth executions have the same timed input sequence but different timed output sequences.

Henceforth the TFSMs-T are complete and initially connected.

2.2 Complete test suite for fault models

Let be a DTFSM-T, called the specification machine.

Definition 3 (Mutation machine for a specification machine)

A nondeterministic TFSM-T is a mutation machine of if is a submachine of . Transitions in but not in or in but not in are called mutated.

A mutant is a deterministic submachine of different from the specification. We let denote the set of mutants in . A mutant represents an implementation of the specification seeded with faults. Faults are represented with mutated transitions and every mutant defines a subset of them. Mutated transitions can represent transfer faults, output faults, changes of timeouts and adding of extra-states.

A transition is suspicious in if defines another transition from the source state of and either both and have the same input or they are timeout transitions. A transition of the specification is called untrusted if it is suspicious in the mutation machine; otherwise, it is trusted. The set of suspicious transitions of is partitioned into a set of untrusted transitions all defined in the specification and the set of mutated transitions undefined in the specification.

Let be a mutant with an initial state of the mutation machine of . We use the state equivalence relation to define conforming mutants.

Definition 4 (Conforming mutants and detected mutants)

Mutant is conforming to , if ; otherwise, it is nonconforming and a timed input sequence such that is said to detect .

We say that mutant survives input sequence if does not detect .

The set of all mutants in mutation machine is called a fault domain for . If is deterministic and complete then is empty. A general fault model is the tuple following [20, 17]. Let denote the set of input/output transitions defined in state with input and denote the set of timeout transitions defined in state . The number of mutants in is given by the formula , where denotes the set of input-output transitions with input defined in and denotes the set of timeout transitions in . The conformance relation partitions the set into conforming mutants and nonconforming ones which we need to detect.

Definition 5 (Complete test suite)

A test for is a timed input sequence. A complete test suite for is a set of test detecting all nonconforming mutants in .

Example 2

In Figure 4, is a mutation machine for the specification machine . and has the same number of states, meaning that the faults represented with mutated transitions in do not introduce extra-states. The mutated transitions are represented by dashed lines. The transitions , and are suspicious; however is not. is trusted and is untrusted. is neither trusted nor untrusted because it does not belong to the specification. defines an output fault on input since the expected output is defined with transition . In state , defines a transfer fault for input and increases the expected timeout for and defines a transfer fault. The transition implements a fault created by reducing the timeout of ; it is defined in the mutant in Figure (c)c. For the timed input sequence , the specification and mutant performs the third and fourth executions in Example 1, respectively. is nonconforming because the produced timed input sequence differ from , the timed output sequence produced by . defines mutants; some of them are conforming and we would like to generate a complete test suite detecting all the nonconforming mutants.

To generate a complete test suite, a test can be computed for each nonconforming mutant by enumerating the mutants one-by-one, which would be inefficient for huge fault domains. We avoid the one-by-one enumeration of the mutants with constraints specifying only test-surviving mutants.

3 Specifying test-surviving mutants

The mutants surviving a test cannot produce any execution with an unexpected timed output sequence for the test. We encode them with Boolean formulas over Boolean transition variables of which the values indicate the presence or absence of transitions of the mutation machine in mutants.

3.1 Revealing combs and involved mutants

The mutants detected by a test exhibit a revealing execution which produces an unexpected timed output sequence and has as the test. Such an execution is permitted by transitions forming a comb-subgraph in the state transition diagram of the mutation machine. Intuitively, a comb for an execution is nothing else but a path augmented with timeout-unexpired transitions, i.e., transitions of which the timeouts have not expired prior to performing an input-output step. These additional timeout transitions are also needed to specify detected mutants and eliminate them from the fault domain. To simplify the notation, we represent comb-subgraphs with sequences of transitions.

A comb of an execution is the sequence of transitions such that permits for every . We say that comb is enabled by the input sequence of . Each timeout or input/output step in is permitted with a unique transition. However, each time-elapsing step is permitted by a timeout transition with an unexpired timeout, i.e., the timeout is not greater than the clock value in the source timed state of the time-elapsing step. So, several combs can permit the same execution since several timeout transitions permit the same time-elapsing step. Note that timeout transitions with finite or infinite timeouts appear in combs when they permit time-elapsing steps preceding input/output steps; later such timeout transitions participate in Boolean encodings of combs involving detected mutants.

Figure 5: A comb for the first execution in Example 1.
Example 3

There are two combs for the first execution in Example 1; this is because the first step of the execution is permitted either by or . The first comb is represented in Figure 5. The timeouts of the transitions represented with vertical arrows have not expired in the execution when the input-output transition is performed. The timeouts of the transitions represented with horizontal arrows have expired. The first comb is deterministic whereas the second comb is nondeterministic because and are two suspicious timeout transitions defined in . The first comb for the first execution is also a comb for the third execution in Example 1; but the second comb is not a comb for the third execution. This is because the application of after time units in is possible if the timeout transition is not passed. The second execution corresponds to a single nondeterministic comb . It is nondeterministic because and are two input/output transitions defined in with the same input . Each occurrence of the timeout transition before an input/output transition indicates that the timeout of has not expired before the input/output transition is passed.

Combs for executions with unexpected timed output sequences reveal nonconforming mutants unless they belong only to nondeterministic submachines. The combs belonging only to nondeterministic submachines have two transitions which are not defined in the same mutant; such combs are called nondeterministic.

Definition 6 (Deterministic and nondeterministic combs)

A comb is nondeterministic if it has two suspicious input-output transitions or two timeout transitions defined in an identical state of the mutation machine; otherwise, it is a deterministic comb.

Clearly, combs in a mutant or the specification are deterministic because two suspicious transitions cannot be defined in an identical state in a mutant or the specification. A nondeterministic submachine of a mutation machine can contain both deterministic and nondeterministic combs.

Figure 6: A fragment of the distinguishing automaton of and with timeouts; is initial.
Definition 7 (Revealing comb)

Let be a comb of an execution from . We say that is -revealing if there exists an execution of such that , while this does not hold for any prefix of . Comb is revealing if it is -revealing for some input sequence .

The specification contains no revealing comb because its executions always produce expected timed output sequences. Only mutants, nondeterministic or incomplete submachines of a mutation machine can contain revealing combs; but mutants contains only deterministic revealing combs.

Let denote the set of deterministic -revealing combs of machine .

Lemma 1

, for any test .


Mutants are deterministic submachines of the mutation machine. Each deterministic revealing comb in is a comb in a mutant, meaning that . Combs in mutants are necessarily deterministic and revealing combs in mutants are deterministic revealing comb in the mutation machine because mutants are deterministic submachines of the mutation machine. Thus

Lemma 2

Let for a test and . is not detected by if and only if .


Assume that is not detected by . Then is either conforming or nonconforming. contains no revealing comb if it is conforming; then for every . If is nonconforming and there is , we get a contradiction with the fact that is not detected by because is the comb for an execution in with timed input sequence and an unexpected timed output sequence. Conversely, by Definition 4 and Definition 7 if then does not detect

Example 4

The comb for the fourth execution in Example 1, namely is revealing and contained in the mutant and mutation machine in Figure 4. To prevent the fourth execution, we must prevent one the transitions in the comb. For example, if we prevent , the other timeout transition defined in will be performed, yielding to the third execution which cannot be performed in the mutant , but rather in . Clearly is not detected by , in the contrary of .

We define a distinguishing automaton with timeouts for the specification and the mutation machine; the automaton define all revealing combs from which we will serve to extract deterministic combs which reveal nonconforming mutants.

Definition 8 (Distinguishing automaton with timeouts)

Given a specification machine and a mutation machine , a finite automaton , where , is the input transition relation, is the timeout transition relation and is the accepting (sink) state, is the distinguishing automaton with timeouts for and , if it holds that:

  • For each and

    • : if there exists s.t.

    • : if there exists , s.t.

  • For each and the only timeout transition defined in the state of the deterministic specification

    • : if there exists s.t. and

    • : if there exists s.t. and and

    • : if there exists s.t. and and

    • : if there exists s.t. and and

    • : if there exists s.t. and and ,
      where if is finite or infinite and .

  • for all and

The seven rules introduce input transitions and timeout transitions in . Each state of is composed of a state of the specification, a state of the mutation machine, the value of the clock of the specification and the value of the clock of the mutation machine. The clock values are needed for selecting timeout transitions. Input transitions are introduced with and . According to , there is a transition to accepting state if different outputs are produced in and for the same input; otherwise the specification and the mutation machine move to next states, as described with rule .
Timeout transitions of the form are introduced by . The value of is the delay for reaching the only timeout defined in from or a timeout defined in from , since multiple timeouts can be defined in states of nondeterministic mutation machines. can be greater than the delays for reaching some timeouts defined in ; however, is never greater than , the delay for reaching the only timeout in . So, if ; a similar statement holds for the clock and a selected timeout transition of the mutation machine. In , both the timeout in and a timeout in expire after time units. In only a timeout defined in expires after time units and the only finite timeout defined in does not expire after time units. A similar phenomenon is described with ; but contrarily to , the only timeout in is and we set the clock of the specification to . Setting the clock to expresses the fact that we do not care any more about finite values of because only the infinite timeout in must be reached. Without this latter abstraction on the values of , the size of could be infinite because we could have to apply infinitely. The rules and are similar to and , except that the only timeout in expires before a timeout in .

Each transition introduced in with , , and , is defined by a transition of the specification and a transition of the mutation machine. A transition introduced with is defined by a timeout transition of the mutation machine and the only timeout transition of the specification in . Every comb of is defined by a comb of the specification and a comb of the mutation machine, i.e., it has been obtained by composing the transitions in a comb of the specification with the transitions in a comb of the mutation machine.

An execution of from a timed state is a sequence of steps between timed states of ; it can be defined similarly to that for a TFSM-T. An execution starting from and ending at is called accepted. As for TFSMs-T, we can associate every execution of with a comb and a timed input sequence. A comb of is accepted if it corresponds to an accepted execution.

Lemma 3

A comb of is revealing if it defines an accepted comb of .


Let be an accepted comb of defined by a deterministic comb of . includes the sink state and it was obtained by composing the transitions in with transitions in the specification with the rules in Definition 8. corresponds to an execution of the mutation machine producing an unexpected timed output sequence, because otherwise would not have been accepted. Consequently is revealing.

A revealing comb can be common to many mutants, in which case those mut