Symbolic Algorithms for Graphs and Markov Decision Processes with Fairness Objectives

03/31/2018 ∙ by Krishnendu Chatterjee, et al. ∙ 0

Given a model and a specification, the fundamental model-checking problem asks for algorithmic verification of whether the model satisfies the specification. We consider graphs and Markov decision processes (MDPs), which are fundamental models for reactive systems. One of the very basic specifications that arise in verification of reactive systems is the strong fairness (aka Streett) objective. Given different types of requests and corresponding grants, the objective requires that for each type, if the request event happens infinitely often, then the corresponding grant event must also happen infinitely often. All ω-regular objectives can be expressed as Streett objectives and hence they are canonical in verification. To handle the state-space explosion, symbolic algorithms are required that operate on a succinct implicit representation of the system rather than explicitly accessing the system. While explicit algorithms for graphs and MDPs with Streett objectives have been widely studied, there has been no improvement of the basic symbolic algorithms. The worst-case numbers of symbolic steps required for the basic symbolic algorithms are as follows: quadratic for graphs and cubic for MDPs. In this work we present the first sub-quadratic symbolic algorithm for graphs with Streett objectives, and our algorithm is sub-quadratic even for MDPs. Based on our algorithmic insights we present an implementation of the new symbolic approach and show that it improves the existing approach on several academic benchmark examples.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

In this work we present faster symbolic algorithms for graphs and Markov decision processes (MDPs) with strong fairness objectives. For the fundamental model-checking problem, the input consists of a model and a specification, and the algorithmic verification problem is to check whether the model satisfies the specification. We first describe the specific model-checking problem we consider and then our contributions.

Models: Graphs and MDPs. Two standard models for reactive systems are graphs and Markov decision processes (MDPs). Vertices of a graph represent states of a reactive system, edges represent transitions of the system, and infinite paths of the graph represent non-terminating trajectories of the reactive system. MDPs extend graphs with probabilistic transitions that represent reactive systems with uncertainty. Thus graphs and MDPs are the de-facto model of reactive systems with nondeterminism, and nondeterminism with stochastic aspects, respectively [18, 3].

Specification: Strong Fairness (aka Streett) Objectives. A basic and fundamental property in the analysis of reactive systems is the strong fairness condition, which informally requires that if events are enabled infinitely often, then they must be executed infinitely often. More precisely, the strong fairness conditions (aka Streett objectives) consist of types of requests and corresponding grants, and the objective requires that for each type if the request happens infinitely often, then the corresponding grant must also happen infinitely often. After safety, reachability, and liveness, the strong fairness condition is one of the most standard properties that arise in the analysis of reactive systems, and chapters of standard textbooks in verification are devoted to it (e.g., [18, Chapter 3.3][31, Chapter 3][2, Chapters 8, 10]). Moreover, all -regular objectives can be described by Streett objectives, e.g., LTL formulas and non-deterministic -automata can be translated to deterministic Streett automata [33] and efficient translation has been an active research area [15, 22, 27]. Thus Streett objectives are a canonical class of objectives that arise in verification.

Satisfaction.

The basic notions of satisfaction for graphs and MDPs are as follows: For graphs the notion of satisfaction requires that there is a trajectory (infinite path) that belongs to the set of paths described by the Streett objective. For MDPs the satisfaction requires that there is a policy to resolve the nondeterminism such that the Streett objective is ensured almost-surely (with probability 1). Thus the algorithmic model-checking problem of graphs and MDPs with Streett objectives is a core problem in verification.

Explicit vs Symbolic Algorithms. The traditional algorithmic studies consider explicit algorithms that operate on the explicit representation of the system. In contrast, implicit or symbolic algorithms only use a set of predefined operations and do not explicitly access the system [19]. The significance of symbolic algorithms in verification is as follows: to combat the state-space explosion, large systems must be succinctly represented implicitly and then symbolic algorithms are scalable, whereas explicit algorithms do not scale as it is computationally too expensive to even explicitly construct the system.

Relevance. In this work we study symbolic algorithms for graphs and MDPs with Streett objectives. Symbolic algorithms for the analysis of graphs and MDPs are at the heart of many state-of-the-art tools such as SPIN, NuSMV for graphs [26, 17] and PRISM, LiQuor, Storm for MDPs [28, 16, 21]. Our contributions are related to the algorithmic complexity of graphs and MDPs with Streett objectives for symbolic algorithms. We first present previous results and then our contributions.

Previous Results. The most basic algorithm for the problem for graphs is based on repeated SCC (strongly connected component) computation, and informally can be described as follows: for a given SCC, (a) if for every request type that is present in the SCC the corresponding grant type is also present in the SCC, then the SCC is identified as “good”, (b) else vertices of each request type that has no corresponding grant type in the SCC are removed, and the algorithm recursively proceeds on the remaining graph. Finally, reachability to good SCCs is computed. The current best-known symbolic algorithm for SCC computation requires symbolic steps, for graphs with vertices [24], and moreover, the algorithm is optimal [14]. For MDPs, the SCC computation has to be replaced by MEC (maximal end-component) computation, and the current best-known symbolic algorithm for MEC computation requires symbolic steps. While there have been several explicit algorithms for graphs with Streett objectives [25, 12], MEC computation [8, 9, 10], and MDPs with Streett objectives [7], as well as symbolic algorithms for MDPs with Büchi objectives [11], the current best-known bounds for symbolic algorithms with Streett objectives are obtained from the basic algorithms, which are for graphs and for MDPs, where is the number of types of request-grant pairs.

Our Contributions. In this work our main contributions are as follows:

  • We present a symbolic algorithm that requires symbolic steps, both for graphs and MDPs, where is the number of edges. In the case , the previous worst-case bounds are quadratic () for graphs and cubic () for MDPs. In contrast, we present the first sub-quadratic symbolic algorithm both for graphs as well as MDPs. Moreover, in practice, since most graphs are sparse (with ), the worst-case bounds of our symbolic algorithm in these cases are . Another interesting contribution of our work is that we also present an symbolic steps algorithm for MEC decomposition, which is relevant for our results as well as of independent interest, as MEC decomposition is used in many other algorithmic problems related to MDPs. Our results are summarized in Table 1.

  • While our main contribution is theoretical, based on the algorithmic insights we also present a new symbolic algorithm implementation for graphs and MDPs with Streett objectives. We show that the new algorithm improves (by around 30%) the basic algorithm on several academic benchmark examples from the VLTS benchmark suite [20].

Symbolic Operations
Problem Basic Algorithm Improved Algorithm Reference
Graphs with Streett Theorem 2
MDPs with Streett Theorem 4
MEC decomposition Theorem 3
Table 1: Symbolic algorithms for Streett objectives and MEC decomposition.

Technical Contributions. The two key technical contributions of our work are as follows:

  • Symbolic Lock Step Search: We search for newly emerged SCCs by a local graph exploration around vertices that lost adjacent edges. In order to find small new SCCs first, all searches are conducted “in parallel”, i.e., in lock-step, and the searches stop as soon as the first one finishes successfully. This approach has successfully been used to improve explicit algorithms [25, 13, 9, 7]. Our contribution is a non-trivial symbolic variant (Section 3) which lies at the core of the theoretical improvements.

  • Symbolic Interleaved MEC Computation: For MDPs the identification of vertices that have to be removed can be interleaved with the computation of MECs such that in each iteration the computation of SCCs instead of MECs is sufficient to make progress [7]. We present a symbolic variant of this interleaved computation. This interleaved MEC computation is the basis for applying the lock-step search to MDPs.

2 Definitions

2.1 Basic Problem Definitions

Markov Decision Processes (MDPs) and Graphs. An MDP  consists of a finite directed graph with a set of vertices  and a set of edges , a partition of the vertices into player 1 vertices and random vertices , and a probabilistic transition function . We call an edge with a player 1 edge and an edge with a random edge. For we define and . The probabilistic transition function is a function from to , where

is the set of probability distributions over

and a random edge if and only if . Graphs are a special case of MDPs with .

Plays and Strategies. A play or infinite path in is an infinite sequence such that for all ; we denote by the set of all plays. A player 1 strategy  is a function that assigns to every finite prefix  of a play that ends in a player 1 vertex  a successor vertex such that ; we denote by the set of all player 1 strategies. A strategy is memoryless if we have for any that end in the same vertex .

Objectives. An objective is a subset of said to be winning for player 1. We say that a play satisfies the objective if . For a vertex set  the reachability objective is the set of infinite paths that contain a vertex of , i.e., . Let for denote the set of vertices that occur infinitely often in . Given a set of pairs of vertex sets with , the Streett objective is the set of infinite paths for which it holds for each that whenever a vertex of occurs infinitely often, then a vertex of occurs infinitely often, i.e., .

Almost-Sure Winning Sets. For any measurable set of plays we denote by the probability that a play starting at belongs to when player 1 plays strategy . A strategy  is almost-sure (a.s.) winning from a vertex for an objective if . The almost-sure winning set of player 1 is the set of vertices for which player 1 has an almost-sure winning strategy. In graphs the existence of an almost-sure winning strategy corresponds to the existence of a play in the objective, and the set of vertices for which player 1 has an (almost-sure) winning strategy is called the winning set of player 1.

Symbolic Encoding of MDPs. Symbolic algorithms operate on sets of vertices, which are usually described by Binary Decision Diagrams (bdds) [29, 1]. In particular Ordered Binary Decision Diagrams [6] (Obdds) provide a canonical symbolic representation of Boolean functions. For the computation of almost-sure winning sets of MDPs it is sufficient to encode MDPs with Obdds and one additional bit that denotes whether a vertex is in  or .

Symbolic Steps. One symbolic step corresponds to one primitive operation as supported by standard symbolic packages like CuDD [34]. In this paper we only allow the same basic set-based symbolic operations as in [32, 23, 5, 11], namely set operations and the following one-step symbolic operations for a set of vertices : (a) the one-step predecessor operator (b) the one-step successor operator and (c) the one-step controllable predecessor operator i.e., the operator computes all vertices such that the successor belongs to  with positive probability. This operator can be defined using the operator and basic set operations as follows: . We additionally allow cardinality computation and picking an arbitrary vertex from a set as in [11].

Symbolic Model. Informally, a symbolic algorithm does not operate on explicit representation of the transition function of a graph, but instead accesses it through and operations. For explicit algorithms, a operation on a set of vertices (resp., a single vertex) requires (resp., the order of indegree/outdegree of the vertex) time. In contrast, for symbolic algorithms operations are considered unit-cost. Thus an interesting algorithmic question is whether better algorithmic bounds can be obtained considering as unit operations. Moreover, the basic set operations are computationally less expensive (as they encode the relationship between the state variables) compared to the symbolic operations (as they encode the transitions and thus the relationship between the present and the next-state variables). In all presented algorithms, the number of set operations is asymptotically at most the number of operations. Hence in the sequel we focus on the number of operations of algorithms.

Algorithmic Problem. Given an MDP  (resp. a graph ) and a set of Streett pairs , the problem we consider asks for a symbolic algorithm to compute the almost-sure winning set (resp. the winning set ), which is also called the qualitative analysis of MDPs (resp. graphs).

2.2 Basic Concepts related to Algorithmic Solution

Reachability. For a graph  and a set of vertices the set is the set of vertices of that can reach a vertex of within , and it can be identified with at most many operations.

Strongly Connected Components. For a set of vertices we denote by the subgraph of the graph  induced by the vertices of . An induced subgraph  is strongly connected if there exists a path in  between every pair of vertices of . A strongly connected component (SCC) of  is a set of vertices  such that the induced subgraph  is strongly connected and is a maximal set in  with this property. We call an SCC trivial if it only contains a single vertex and no edges; and non-trivial otherwise. The SCCs of  partition its vertices and can be found in symbolic steps [24]. A bottom SCC  in a directed graph  is an SCC with no edges from vertices of  to vertices of , i.e., an SCC without outgoing edges. Analogously, a top SCC  is an SCC with no incoming edges from . For more intuition for bottom and top SCCs, consider the graph in which each SCC is contracted into a single vertex (ignoring edges within an SCC). In the resulting directed acyclic graph the sinks represent the bottom SCCs and the sources represent the top SCCs. Note that every graph has at least one bottom and at least one top SCC. If the graph is not strongly connected, then there exist at least one top and at least one bottom SCC that are disjoint and thus one of them contains at most half of the vertices of .

Random Attractors. In an MDP  the random attractor of a set of vertices is defined as where and for all . The attractor can be computed with at most many operations.

Maximal End-Components. Let be a vertex set without outgoing random edges, i.e., with for all . A sub-MDP of an MDP  induced by a vertex set without outgoing random edges is defined as . Note that the requirement that has no outgoing random edges is necessary in order to use the same probabilistic transition function . An end-component (EC) of an MDP  is a set of vertices such that (a) has no outgoing random edges, i.e., is a valid sub-MDP, (b) the induced sub-MDP is strongly connected, and (c) contains at least one edge. Intuitively, an end-component is a set of vertices for which player 1 can ensure that the play stays within the set and almost-surely reaches all the vertices in the set (infinitely often). An end-component is a maximal end-component (MEC) if it is maximal under set inclusion. An end-component is trivial if it consists of a single vertex (with a self-loop), otherwise it is non-trivial. The MEC decomposition of an MDP consists of all MECs of the MDP.

Good End-Components. All algorithms for MDPs with Streett objectives are based on finding good end-components, defined below. Given the union of all good end-components, the almost-sure winning set is obtained by computing the almost-sure winning set for the reachability objective with the union of all good end-components as the target set. The correctness of this approach is shown in [7, 30] (see also [3, Chap. 10.6.3]). For Streett objectives a good end-component is defined as follows. In the special case of graphs they are called good components.

Definition 1 (Good end-component)

Given an MDP and a set of target pairs, a good end-component is an end-component of such that for each either or . A maximal good end-component is a good end-component that is maximal with respect to set inclusion.

Lemma 1 (Correctness of Computing Good End-Components [30, Corollary 2.6.5, Proposition 2.6.9])

For an MDP  and a set  of target pairs, let be the set of all maximal good end-components. Then is equal to .

Iterative Vertex Removal. All the algorithms for Streett objectives maintain vertex sets that are candidates for good end-components. For such a vertex set  we (a) refine the maintained sets according to the SCC decomposition of and (b) for a set of vertices  for which we know that it cannot be contained in a good end-component, we remove its random attractor from . The following lemma shows the correctness of these operations.

Lemma 2 (Correctness of Vertex Removal [30, Lemma 2.6.10])

Given an MDP , let be an end-component with for some . Then

  • for one SCC  of and

  • for each and each sub-MDP  containing .

Let be a good end-component. Then is an end-component and for each index , implies . Hence we obtain the following corollary.

Corollary 1 ([30, Corollary 4.2.2])

Given an MDP , let be a good end-component with for some . For each with it holds that .

For an index  with we call the vertices of bad vertices. The set of all bad vertices can be computed with set operations.

3 Symbolic Divide-and-Conquer with Lock-Step Search

In this section we present a symbolic version of the lock-step search for strongly connected subgraphs [25]. This symbolic version is used in all subsequent results, i.e., the sub-quadratic symbolic algorithms for graphs and MDPs with Streett objectives, and for MEC decomposition.

Divide-and-Conquer. The common property of the algorithmic problems we consider in this work is that the goal is to identify subgraphs of the input graph that are strongly connected and satisfy some additional properties. The difference between the problems lies in the required additional properties. We describe and analyze the Procedure LABEL:proc:lockstep that we use in all our improved algorithms to efficiently implement a divide-and-conquer approach based on the requirement of strong connectivity, that is, we divide a subgraph , induced by a set of vertices , into two parts that are not strongly connected within or detect that is strongly connected.

Start Vertices of Searches. The input to Procedure LABEL:proc:lockstep is a set of vertices  and two subsets of denoted by  and . In the algorithms that call the procedure as a subroutine, vertices contained in  have lost incoming edges (i.e., they were a “head” of a lost edge) and vertices contained in  have lost outgoing edges (i.e., they were a “tail” of a lost edge) since the last time a superset of was identified as being strongly connected. For each vertex  of the procedure conducts a backward search (i.e., a sequence of operations) within  to find the vertices of that can reach ; and analogously a forward search (i.e., a sequence of operations) from each vertex  of is conducted.

Intuition for the Choice of Start Vertices. If the subgraph  is not strongly connected, then it contains at least one top SCC and at least one bottom SCC that are disjoint. Further, if for a superset the subgraph  was strongly connected, then each top SCC of contains a vertex that had an additional incoming edge in compared to , and analogously each bottom SCC of contains a vertex that had an additional outgoing edge. Thus by keeping track of the vertices that lost incoming or outgoing edges, the following invariant will be maintained by all our improved algorithms.

Invariant 1 (Start Vertices Sufficient)

We have . Either (a) and is strongly connected or (b) at least one vertex of each top SCC of is contained in and at least one vertex of each bottom SCC of is contained in .

algocf[htbp]    

Lock-Step Search. The searches from the vertices of are performed in lock-step, that is, (a) one step is performed in each of the searches before the next step of any search is done and (b) all searches stop as soon as the first of the searches finishes. This is implemented in Procedure LABEL:proc:lockstep as follows. A step in the search from a vertex (and analogously for ) corresponds to the execution of the iteration of the for-each loop for . In an iteration of a for-each loop we might discover that we do not need to consider this search further (see the paragraph on ensuring strong connectivity below) and update the set (via ) for future iterations accordingly. Otherwise the set is either strictly increasing in this step of the search or the search for terminates and we return the set of vertices in that are reachable from . So the two for-each loops over the vertices of and that are executed in an iteration of the while-loop perform one step of each of the searches and the while-loop stops as soon as a search stops, i.e., a return statement is executed and hence this implements properties (a) and (b) of lock-step search. Note that the while-loop terminates, i.e., a return statement is executed eventually because for all (and resp. for all ) the sets are monotonically increasing over the iterations of the while-loop, we have , and if some set does not increase in an iteration, then it is either removed from and thus not considered further or a return statement is executed. Note that when a search from a vertex stops, it has discovered a maximal set of vertices  that can be reached from ; and analogously for . Figure 1 shows a small intuitive example of a call to the procedure.

Figure 1: An example of symbolic lock-step search showing the first three iterations of the main while-loop. Note that during the second iteration, the search started from is disregarded since it collides with . In the subsequent fourth iteration, the search started from is returned by the procedure.

Comparison to Explicit Algorithm. In the explicit version of the algorithm [25, 7] the search from vertex  performs a depth-first search that terminates exactly when every edge reachable from is explored. Since any search that starts outside of a bottom SCC but reaches the bottom SCC has to explore more edges than the search started inside of the bottom SCC, the first search from a vertex of that terminates has exactly explored (one of) the smallest (in the number of edges) bottom SCC(s) of . Thus on explicit graphs the explicit lock-step search from the vertices of finds (one of) the smallest (in the number of edges) top or bottom SCC(s) of in time proportional to the number of searches times the number of edges in the identified SCC. In symbolically represented graphs it can happen (1) that a search started outside of a bottom (resp. top) SCC terminates earlier than the search started within the bottom (resp. top) SCC and (2) that a search started in a larger (in the number of vertices) top or bottom SCC terminates before one in a smaller top or bottom SCC. We discuss next how we address these two challenges.

Ensuring Strong Connectivity. First, we would like the set returned by Procedure LABEL:proc:lockstep to indeed be a top or bottom SCC of . For this we use the following observation for bottom SCCs that can be applied to top SCCs analogously. If a search starting from a vertex of encounters another vertex , , there are two possibilities: either (1) both vertices are in the same SSC or (2) can reach but not vice versa. In Case (1) the searches from both vertices can explore all vertices in the SCC and thus it is sufficient to only search from one of them. In Case (2) the SCC of  has an outgoing edge and thus cannot be a bottom SCC. Hence in both cases we can remove the vertex from the set while still maintaining Invariant 1. By Invariant 1 we further have that each search from a vertex of that is not in a bottom SCC encounters another vertex of in its search and therefore is removed from the set during Procedure LABEL:proc:lockstep (if no top or bottom SCC is found earlier). This ensures that the returned set is either a top or a bottom SCC.111To improve the practical performance, we return the updated sets and . By the above argument this preserves Invariant 1.

Bound on Symbolic Steps. Second, observe that we can still bound the number of symbolic steps needed for the search that terminates first by the number of vertices in the smallest top or bottom SCC of , since this is an upper bound on the symbolic steps needed for the search started in this SCC. Thus provided Invariant 1, we can bound the number of symbolic steps in Procedure LABEL:proc:lockstep to identify a vertex set such that and are not strongly connected in by . In the algorithms that call Procedure LABEL:proc:lockstep we charge the number of symbolic steps in the procedure to the vertices in the smaller set of and ; this ensures that each vertex is charged at most times over the whole algorithm. We obtain the following result (proof in Appendix 0.A).

Theorem 1 (Lock-Step Search)

Provided Invariant 1 holds, Procedure LABEL:proc:lockstep(, , , ) returns a top or bottom SCC  of . It uses symbolic steps if and otherwise.

4 Graphs with Streett Objectives

Basic Symbolic Algorithm. Recall that for a given graph (with vertices) and a Streett objective (with target pairs) each non-trivial strongly connected subgraph without bad vertices is a good component. The basic symbolic algorithm for graphs with Streett objectives repeatedly removes bad vertices from each SCC and then recomputes the SCCs until all good components are found. The winning set then consists of the vertices that can reach a good component. We refer to this algorithm as LABEL:alg:streettgraphbasic. For the pseudocode and more details see Appendix 0.B.

Proposition 1

Algorithm LABEL:alg:streettgraphbasic correctly computes the winning set in graphs with Streett objectives and requires symbolic steps.

Improved Symbolic Algorithm. In our improved symbolic algorithm we replace the recomputation of all SCCs with the search for a new top or bottom SCC with Procedure LABEL:proc:lockstep from vertices that have lost adjacent edges whenever there are not too many such vertices. We present the improved symbolic algorithm for graphs with Streett objectives in more detail as it also conveys important intuition for the MDP case. The pseudocode is given in Algorithm LABEL:alg:streettgraphimpr.

Iterative Refinement of Candidate Sets. The improved algorithm maintains a set goodC of already identified good components that is initially empty and a set  of candidates for good components that is initialized with the SCCs of the input graph . The difference to the basic algorithm lies in the properties of the vertex sets maintained in and the way we identify sets that can be separated from each other without destroying a good component. In each iteration one vertex set  is removed from and, after the removal of bad vertices from the set, either identified as a good component or split into several candidate sets. By Lemma 2 and Corollary 1 the following invariant is maintained throughout the algorithm for the sets in goodC and .

Invariant 2 (Maintained Sets)

The sets in are pairwise disjoint and for every good component  of there exists a set such that either or .

Lost Adjacent Edges. In contrast to the basic algorithm, the subgraph induced by a set  contained in  is not necessarily strongly connected. Instead, we remember vertices of  that have lost adjacent edges since the last time a superset of was determined to induce a strongly connected subgraph; vertices that lost incoming edges are contained in  and vertices that lost outgoing edges are contained in . In this way we maintain Invariant 1 throughout the algorithm, which enables us to use Procedure LABEL:proc:lockstep with the running time guarantee provided by Theorem 1.

algocf[t!]    

Identifying SCCs. Let be the vertex set removed from in a fixed iteration of Algorithm LABEL:alg:streettgraphimpr after the removal of bad vertices in the inner while-loop. First note that if is strongly connected and contains at least one edge, then it is a good component. If the set was already identified as strongly connected in a previous iteration, i.e., and are empty, then is identified as a good component in line LABEL:l:gimpr:good1. If many vertices of have lost adjacent edges since the last time a super-set of was identified as a strongly connected subgraph, then the SCCs of are determined as in the basic algorithm. To achieve the optimal asymptotic upper bound, we say that many vertices of have lost adjacent edges when we have , while lower thresholds are used in our experimental results. Otherwise, if not too many vertices of  lost adjacent edges, then we start a symbolic lock-step search for top SCCs from the vertices of  and for bottom SCCs from the vertices of  using Procedure LABEL:proc:lockstep. The set returned by the procedure is either a top or a bottom SCC of (Theorem 1). Therefore we can from now on consider and separately, maintaining Invariants 1 and 2.

Algorithm LABEL:alg:streettgraphimpr. A succinct description of the pseudocode is as follows: Lines LABEL:l:gimpr:initstartLABEL:l:gimpr:initend initialize the set of candidates for good components with the SCCs of the input graph. In each iteration of the main while-loop one candidate is considered and the following operations are performed: (a) lines LABEL:l:gimpr:badstartLABEL:l:gimpr:badend iteratively remove all bad vertices; if afterwards the candidate is still strongly connected (and contains at least one edge), it is identified as a good component in the next step; otherwise it is partitioned into new candidates in one of the following ways: (b) if many vertices lost adjacent edges, lines LABEL:l:gimpr:basicstartLABEL:l:gimpr:basicend partition the candidate into its SCCs (this corresponds to an iteration of the basic algorithm); (c) otherwise, lines LABEL:l:gimpr:lssbegLABEL:l:gimpr:lssend use symbolic lock-step search to partition the candidate into one of its SCCs and the remaining vertices. The while-loop terminates when no candidates are left. Finally, vertices that can reach some good component are returned. We have the following result (proof in Appendix 0.B).

Theorem 2 (Improved Algorithm for Graphs)

Algorithm LABEL:alg:streettgraphimpr correctly computes the winning set in graphs with Streett objectives and requires symbolic steps.

5 Symbolic MEC Decomposition

In this section we present a succinct description of the basic symbolic algorithm for MEC decomposition and then present the main ideas for the improved algorithm.

Basic symbolic algorithm for MEC decomposition. The basic symbolic algorithm for MEC decomposition maintains a set of identified MECs and a set of candidates for MECs, initialized with the SCCs of the MDP. Whenever a candidate is considered, either (a) it is identified as a MEC or (b) it contains vertices with outgoing random edges, which are then removed together with their random attractor from the candidate, and the SCCs of the remaining sub-MDP are added to the set of candidates. We refer to the algorithm as LABEL:alg:mecbasic.

Proposition 2

Algorithm LABEL:alg:mecbasic correctly computes the MEC decomposition of MDPs and requires symbolic steps.

Improved symbolic algorithm for MEC decomposition. The improved symbolic algorithm for MEC decomposition uses the ideas of symbolic lock-step search presented in Section 3. Informally, when considering a candidate that lost a few edges from the remaining graph, we use the symbolic lock-step search to identify some bottom SCC. We refer to the algorithm as LABEL:alg:mecimpr. Since all the important conceptual ideas regarding the symbolic lock-step search are described in Section 3, we relegate the technical details to Appendix 0.C. We summarize the main result (proof in Appendix 0.C).

Theorem 3 (Improved Algorithm for MEC)

Algorithm LABEL:alg:mecimpr correctly computes the MEC decomposition of MDPs and requires symbolic steps.

6 MDPs with Streett Objectives

Basic Symbolic Algorithm. We refer to the basic symbolic algorithm for MDPs with Streett objectives as LABEL:alg:streettmdpbasic, which is similar to the algorithm for graphs, with SCC computation replaced by MEC computation. The pseudocode of Algorithm LABEL:alg:streettmdpbasic together with its detailed description is presented in Appendix 0.D.

Proposition 3

Algorithm LABEL:alg:streettmdpbasic correctly computes the almost-sure winning set in MDPs with Streett objectives and requires symbolic steps.

Remark

The above bound uses the basic symbolic MEC decomposition algorithm. Using our improved symbolic MEC decomposition algorithm, the above bound could be improved to .

Improved Symbolic Algorithm. We refer to the improved symbolic algorithm for MDPs with Streett objectives as LABEL:alg:streettmdpimpr. First we present the main ideas for the improved symbolic algorithm. Then we explain the key differences compared to the improved symbolic algorithm for graphs. A thorough description with the technical details and proofs is presented in Appendix 0.D.

  • First, we improve the algorithm by interleaving the symbolic MEC computation with the detection of bad vertices [7, 30]. This allows to replace the computation of MECs in each iteration of the while-loop with the computation of SCCs and an additional random attractor computation.

    • Intuition of interleaved computation. Consider a candidate for a good end-component after a random attractor to some bad vertices is removed from it. After the removal of the random attractor, the set does not have random vertices with outgoing edges. Consider that further holds. If is strongly connected and contains an edge, then it is a good end-component. If is not strongly connected, then contains at least two SCCs and some of them might have random vertices with outgoing edges. Since end-components are strongly connected and do not have random vertices with outgoing edges, we have that (1) every good end-component is completely contained in one of the SCCs of and (2) the random vertices of an SCC with outgoing edges and their random attractor do not intersect with any good end-component (see Lemma 2).

    • Modification from basic to improved algorithm. We use these observations to modify the basic algorithm as follows: First, for the sets that are candidates for good end-components, we do not maintain the property that they are end-components, but only that they do not have random vertices with outgoing edges (it still holds that every maximal good end-component is either already identified or contained in one of the candidate sets). Second, for a candidate set , we repeat the removal of bad vertices until holds before we continue with the next step of the algorithm. This allows us to make progress after the removal of bad vertices by computing all SCCs (instead of MECs) of the remaining sub-MDP. If there is only one SCC, then this is a good end-component (if it contains at least one edge). Otherwise (a) we remove from each SCC the set of random vertices with outgoing edges and their random attractor and (b) add the remaining vertices of each SCC as a new candidate set.

  • Second, as for the improved symbolic algorithm for graphs, we use the symbolic lock-step search to quickly identify a top or bottom SCC every time a candidate has lost a small number of edges since the last time its superset was identified as being strongly connected. The symbolic lock-step search is described in detail in Section 3.

Using interleaved MEC computation and lock-step search leads to a similar algorithmic structure for Algorithm LABEL:alg:streettmdpimpr as for our improved symbolic algorithm for graphs (Algorithm LABEL:alg:streettgraphimpr). The key differences are as follows: First, the set of candidates for good end-components is initialized with the MECs of the input graph instead of the SCCs. Second, whenever bad vertices are removed from a candidate, also their random attractor is removed. Further, whenever a candidate is partitioned into its SCCs, for each SCC, the random attractor of the vertices with outgoing random edges is removed. Finally, whenever a candidate is separated into and via symbolic lock-step search, the random attractor of the vertices with outgoing random edges is removed from , and the random attractor of is removed from .

Theorem 4 (Improved Algorithm for MDPs)

Algorithm LABEL:alg:streettmdpimpr correctly computes the almost-sure winning set in MDPs with Streett objectives and requires symbolic steps.

7 Experiments

We present a basic prototype implementation of our algorithm and compare against the basic symbolic algorithm for graphs and MDPs with Streett objectives.

Models. We consider the academic benchmarks from the VLTS benchmark suite [20], which gives representative examples of systems with nondeterminism, and has been used in previous experimental evaluation (such as [4, 11]).

Specifications. We consider random LTL formulae and use the tool Rabinizer [27] to obtain deterministic Rabin automata. Then the negations of the formulae give us Streett automata, which we consider as the specifications.

Graphs. For the models of the academic benchmarks, we first compute SCCs, as all algorithms for Streett objectives compute SCCs as a preprocessing step. For SCCs of the model benchmarks we consider products with the specification Streett automata, to obtain graphs with Streett objectives, which are the benchmark examples for our experimental evaluation. The number of transitions in the benchmarks ranges from K to Million.

MDPs. For MDPs, we consider the graphs obtained as above and consider a fraction of the vertices of the graph as random vertices, which is chosen uniformly at random. We consider , , and of the vertices as random vertices for different experimental evaluation.

Figure 2: Results for graphs with Streett objectives.
(a) 10% random vertices
(b) 20% random vertices
(c) 50% random vertices
Figure 6: Results for MDPs with Streett objectives.

Experimental evaluation. In the experimental evaluation we compare the number of symbolic steps (i.e., the number of operations222Recall that the basic set operations are cheaper to compute, and asymptotically at most the number of operations in all the presented algorithms.) executed by the algorithms, the comparison of running time yields similar results and is provided in Appendix 0.E. As the initial preprocessing step is the same for all the algorithms (computing all SCCs for graphs and all MECs for MDPs), the comparison presents the number of symbolic steps executed after the preprocessing. The experimental results for graphs are shown in Figure 2 and the experimental results for MDPs are shown in Figure 6 (in each figure the two lines represent equality and an order-of-magnitude improvement, respectively).

Discussion. Note that the lock-step search is the key reason for theoretical improvement, however, the improvement relies on a large number of Streett pairs. In the experimental evaluation, the LTL formulae generate Streett automata with small number of pairs, which after the product with the model accounts for an even smaller fraction of pairs as compared to the size of the state space. This has two effects:

  • In the experiments the lock-step search is performed for a much smaller parameter value ( instead of the theoretically optimal bound of ), and leads to a small improvement.

  • For large graphs, since the number of pairs is small as compared to the number of states, the improvement over the basic algorithm is minimal.

In contrast to graphs, in MDPs even with small number of pairs as compared to the state-space, the interleaved MEC computation has a notable effect on practical performance, and we observe performance improvement even in large MDPs.

8 Conclusion

In this work we consider symbolic algorithms for graphs and MDPs with Streett objectives, as well as for MEC decomposition. Our algorithmic bounds match for both graphs and MDPs. In contrast, while SCCs can be computed in linearly many symbolic steps no such algorithm is known for MEC decomposition. An interesting direction of future work would be to explore further improved symbolic algorithms for MEC decomposition. Moreover, further improved symbolic algorithms for graphs and MDPs with Streett objectives is also an interesting direction of future work.

Acknowledgements.

K. C. and M. H. are partially supported by the Vienna Science and Technology Fund (WWTF) grant ICT15-003. K. C. is partially supported by the Austrian Science Fund (FWF): S11407-N23 (RiSE/SHiNE), and an ERC Start Grant (279307: Graph Games). V. T.  is partially supported by the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie Grant Agreement No. 665385. V. L. is partially supported by the Austrian Science Fund (FWF): S11408-N23 (RiSE/SHiNE), the ISF grant #1278/16, and an ERC Consolidator Grant (project MPM). For M. H. and V. L. the research leading to these results has received funding from the European Research Council under the European Union’s Seventh Framework Programme (FP/2007-2013) / ERC Grant Agreement no. 340506.

References

  • [1] Akers, S.B.: Binary decision diagrams. IEEE Trans. Comput. C-27(6), 509–516 (1978)
  • [2] Alur, R., Henzinger, T.A.: Computer-aided verification (2004), http://www.cis.upenn.edu/group/cis673/
  • [3] Baier, C., Katoen, J.P.: Principles of model checking. MIT Press (2008)
  • [4] Barnat, J., Chaloupka, J., van de Pol, J.: Distributed algorithms for SCC decomposition. J. Log. Comput. 21(1), 23–44 (2011)
  • [5] Bloem, R., Gabow, H.N., Somenzi, F.: An algorithm for strongly connected component analysis in n log n symbolic steps. Form. Methods Syst. Des. 28(1), 37–56 (2006)
  • [6] Bryant, R.E.: Symbolic manipulation of Boolean functions using a graphical representation. In: Conference on Design automation (DAC). pp. 688–694 (1985)
  • [7] Chatterjee, K., Dvořák, W., Henzinger, M., Loitzenbauer, V.: Model and objective separation with conditional lower bounds: Disjunction is harder than conjunction. In: LICS. pp. 197–206 (2016)
  • [8] Chatterjee, K., Henzinger, M.: Faster and Dynamic Algorithms For Maximal End-Component Decomposition And Related Graph Problems In Probabilistic Verification. In: SODA. pp. 1318–1336 (2011)
  • [9] Chatterjee, K., Henzinger, M.: An Time Algorithm for Alternating Büchi Games. In: SODA. pp. 1386–1399 (2012)
  • [10] Chatterjee, K., Henzinger, M.: Efficient and Dynamic Algorithms for Alternating Büchi Games and Maximal End-Component Decomposition. Journal of the ACM 61(3),  15 (2014)
  • [11] Chatterjee, K., Henzinger, M., Joglekar, M., Shah, N.: Symbolic algorithms for qualitative analysis of Markov decision processes with Büchi objectives. Form. Methods Syst. Des. 42(3), 301–327 (2013)
  • [12] Chatterjee, K., Henzinger, M., Loitzenbauer, V.: Improved Algorithms for One-Pair and -Pair Streett Objectives. In: LICS. pp. 269–280 (2015)
  • [13] Chatterjee, K., Jurdziński, M., Henzinger, T.A.: Simple stochastic parity games. In: CSL. pp. 100–113 (2003)
  • [14] Chatterjee, K., Dvořák, W., Henzinger, M., Loitzenbauer, V.: Lower bounds for symbolic computation on graphs: Strongly connected components, liveness, safety, and diameter. In: SODA. pp. 2341–2356 (2018)
  • [15] Chatterjee, K., Gaiser, A., Kretínský, J.: Automata with generalized rabin pairs for probabilistic model checking and LTL synthesis. In: CAV. pp. 559–575 (2013)
  • [16] Ciesinski, F., Baier, C.: LiQuor: A tool for qualitative and quantitative linear time analysis of reactive systems. In: QEST. pp. 131–132 (2006)
  • [17] Cimatti, A., Clarke, E., Giunchiglia, F., Roveri, M.: NUSMV: a new symbolic model checker. International Journal on Software Tools for Technology Transfer (STTT) 2(4), 410–425 (2000)
  • [18] Clarke, Jr., E.M., Grumberg, O., Peled, D.A.: Model Checking. MIT Press, Cambridge, MA, USA (1999)
  • [19] Clarke, E., Grumberg, O., Peled, D.: Symbolic model checking. In: Model Checking. MIT Press (1999)
  • [20] CWI/SEN2 and INRIA/VASY: The VLTS Benchmark Suite, http://cadp.inria.fr/resources/vlts
  • [21] Dehnert, C., Junges, S., Katoen, J., Volk, M.: A Storm is coming: A modern probabilistic model checker. In: CAV. pp. 592–600 (2017)
  • [22] Esparza, J., Kretínský, J.: From LTL to deterministic automata: A safraless compositional approach. In: CAV. pp. 192–208 (2014)
  • [23] Gentilini, R., Piazza, C., Policriti, A.: Computing strongly connected components in a linear number of symbolic steps. In: SODA. pp. 573–582 (2003)
  • [24] Gentilini, R., Piazza, C., Policriti, A.: Symbolic graphs: Linear solutions to connectivity related problems. Algorithmica 50(1), 120–158 (2008)
  • [25] Henzinger, M., Telle, J.A.: Faster Algorithms for the Nonemptiness of Streett Automata and for Communication Protocol Pruning. In: SWAT. pp. 16–27 (1996)
  • [26] Holzmann, G.J.: The model checker SPIN. IEEE Trans. Softw. Eng. 23(5), 279–295 (1997)
  • [27] Komárková, Z., Kretínský, J.: Rabinizer 3: Safraless translation of LTL to small deterministic automata. In: ATVA. pp. 235–241 (2014)
  • [28] Kwiatkowska, M.Z., Norman, G., Parker, D.: PRISM 4.0: Verification of probabilistic real-time systems. In: CAV. pp. 585–591 (2011)
  • [29] Lee, C.Y.: Representation of switching circuits by binary-decision programs. Bell System Techn. J. 38(4), 985–999 (1959)
  • [30] Loitzenbauer, V.: Improved Algorithms and Conditional Lower Bounds for Problems in Formal Verification and Reactive Synthesis. Ph.D. thesis, University of Vienna (2016)
  • [31] Manna, Z., Pnueli, A.: Temporal Verification of Reactive Systems: Progress (Draft) (1996)
  • [32] Ravi, K., Bloem, R., Somenzi, F.: A comparative study of symbolic algorithms for the computation of fair cycles. In: FMCAD. pp. 143–160 (2000)
  • [33] Safra, S.: On the complexity of -automata. In: FOCS. pp. 319–327 (1988)
  • [34] Somenzi, F.: CUDD: CU decision diagram package release 3.0.0 (2015), http://vlsi.colorado.edu/~fabio/CUDD/

Appendix

Appendix 0.A Details of Section 3: Symbolic Lock-Step Search

Proof (of Theorem 1)

Strong connectivity. We want to show that LABEL:proc:lockstep(, , , ) is a top or bottom SCC of given Invariant 1 is satisfied. By the invariant at least one vertex of each top SCC of is contained in  and at least one vertex of each bottom SCC of is contained in . Suppose is the set obtained from a search conducted by operations that started from within a bottom SCC of . Since is a bottom SCC and we update the search by executing operations (and moreover intersect with at every update), we have . Further, since is an SCC, the updates with eventually cover all vertices of , which gives us . A set  constructed with operations whose start vertex  is not contained in a bottom SCC of can not yield the set since eventually it contains a bottom SCC of , and by Invariant 1 this SCC contains a candidate in ; therefore is satisfied at some point in the construction of and then search is canceled by removing from ; note that a search starting from a bottom SCC can be canceled only if another vertex of the bottom SCC remains in . By the symmetric argument for searches conducted by operations that started from a vertex of a top SCC we have that the returned set is either a top or a bottom SCC of .

Bound on symbolic steps. Consider (one of) the smallest top or bottom SCCs of . Suppose w.l.o.g. that is a bottom SCC. By Invariant 1 there is a search, conducted by operations, that starts from a vertex within and that is not canceled, and therefore this search terminates after at most many operations. Other searches may terminate earlier but this gives an upper bound of on the number of symbolic steps until the lock-step search terminates. Finally, consider the returned set LABEL:proc:lockstep(, , , ). There are two possible cases: either (i) , which implies so the number of symbolic steps can be bounded by , or (ii) . In the second case, since is (some) smallest SCC, is an SCC, and contains at least one SCC, we have and , and hence we can bound the number of symbolic steps in this case by . ∎

Appendix 0.B Details of Section 4: Graphs with Streett Objectives

0.b.1 Basic Symbolic Algorithm for Graphs with Streett Objectives

algocf[htbp]    

The pseudocode of the basic symbolic algorithm for graphs with Streett objectives is given in Algorithm LABEL:alg:streettgraphbasic.

The basic symbolic algorithm for Streett objectives on graphs finds good components as follows. The algorithm maintains two sets of vertex sets: goodC contains identified good components and is initially empty; contains candidates for good components and is initialized with the SCCs of the input graph . The sets in are strongly connected subgraphs of  throughout the algorithm. In each iteration of the while-loop one of the candidate sets  maintained in is considered. If the set  does not contain bad vertices and contains at least one edge, then it is a good component and added to goodC. Otherwise, the set of bad vertices  in  is removed from ; the subgraph induced by might not be strongly connected but every good component contained in must still be strongly connected, therefore the maximal strongly connected subgraphs of are added to as new candidates for good components. By Lemma 2 and Corollary 1 this procedure maintains the property that every good component of  is completely contained in one of the vertex sets of goodC or . Further in each iteration either (a) vertices are removed or separated into different vertex sets or (b) a new good component is identified. Thus after at most iterations the set is empty and all good components of  are contained in goodC. Furthermore, whenever bad vertices are removed from a given candidate set, the number of target pairs this candidate set intersects is reduced by one. Thus each vertex is considered in at most iterations of the main while-loop. Finally, the set of vertices that can reach a good component is determined (by operations) and output as the winning set. Since computing SCCs can be done in symbolic steps, the total number of symbolic steps of the basic algorithm is bounded by .

0.b.2 Improved Symbolic Algorithm for Graphs with Streett Objectives

Lemma 3 (Invariants of Improved Algorithm for Graphs)

Invariant 1 and Invariant 2 are preserved throughout Algorithm LABEL:alg:streettgraphimpr, i.e., they hold before the first iteration, after each iteration, and after termination of the main while-loop. Further, Invariant 1 is preserved during each iteration of the main while-loop.

Proof

Invariant 1. Whenever a new candidate is added as a result from allSCCs, it is strongly connected, and we set ; this in particular implies that the invariant is satisfied after the initialization of the algorithm.

By induction and Theorem 1, the invariant is satisfied whenever Procedure LABEL:proc:lockstep returns a candidate and we set .

Now consider an update of a candidate where some subset is deleted from it and assume the invariant holds before the update. In these cases we update and by setting and . This adds the vertices that remain in  and have an edge from a vertex of to and those with an edge to to . Suppose a new top (resp. bottom) SCC emerges in by the removal of from . Then some vertex of had an outgoing edge to (resp. an incoming edge from ) and thus is contained in the updated set (resp. ), maintaining the invariant. This happens whenever we remove from , and whenever we subtract a result from Procedure LABEL:proc:lockstep from .

Invariant 2 – Disjointness. The sets in are pairwise disjoint at the initialization since goodC is initialized as . Furthermore, whenever a set is added to goodC in an iteration of the main while-loop, a superset is removed from in the same iteration of the while-loop. Therefore by induction the disjointness of the sets in is preserved.

Invariant 2 – Containment of good components. At initialization, contains all SCCs of the input graph . Each good component of is strongly connected, so there exists an SCC such that for each good component .

Consider a set that is removed from at the beginning of an iteration of the main while-loop. Consider further a good component of such that . We require that a set is added to either or goodC in this iteration of the main while-loop.

First, whenever we remove from , by Corollary 1 we maintain the fact that . Second, contains an edge since . Finally, one of the three cases happens:

Case (1): If , then the set is added to goodC.

Case (2): If , then the algorithm computes the SCCs of . Since is strongly connected, it is completely contained in some SCC of , and is added either to or to goodC.

Case (3): If , then the algorithm either adds to goodC, or partitions into and . Suppose the latter case happens, then by Theorem 1 we have that is an SCC of . Further, since is strongly connected, it is completely contained in some SCC of . Therefore either or , and both and are added to .

By the above case analysis we have that a set is added to either or goodC in the iteration of the main while-loop, and thus the invariant is preserved throughout the algorithm. ∎

Proof (of Theorem 2)

Correctness. Whenever a candidate set is added to goodC, it contains an edge by the check at line LABEL:l:p1, and by the check at line LABEL:l:gimpr:innerw. Furthermore, (a) at line LABEL:l:gimpr:good1, is strongly connected by Invariant 1, (b) at line LABEL:l:gimpr:good2, is strongly connected by the result of allSCCs, and (c) at line LABEL:l:gimpr:good3, is strongly connected by Theorem 1. Therefore we have that whenever a candidate set is added to goodC, it is indeed a good component (soundness).

Finally, by soundness, Invariant 2, the termination of the algorithm (shown below), and the fact that at the termination of the algorithm, we have that goodC contains all good components of (completeness).

Symbolic steps analysis. By [24], the initialization with the SCCs of the input graph takes symbolic steps. Furthermore, the reachability computation in the last step takes operations.

In each iteration of the outer while-loop, a set  is removed from and either (a) a set is added to goodC and no set is added to or (b) at least two sets that are (proper subsets of) a partition of are added to . Both can happen at most times, thus there can be at most iterations of the outer while-loop. The and operations at lines LABEL:l:p1LABEL:l:p2, and LABEL:l:p3 can be charged to the iterations of the outer while-loop.

An iteration of the inner while-loop (lines LABEL:l:gimpr:innerw-LABEL:l:gimpr:badend) is executed only if some vertices  are removed from ; the vertices of are then not considered further. Thus there can, in total, be at most and operations over all iterations of the inner while-loop.

Note that every vertex in each of and can be attributed to at least one unique implicit edge deletion since we only add vertices to resp.  that are successors resp. predecessors of vertices that were separated from (or deleted from the maintained graph). Whenever the case occurs, for all subsets that are then added to , we initialize . Therefore the case can happen at most times throughout the algorithm since there are at most edges that can be deleted, and hence in total takes symbolic steps.

It remains to bound the number of symbolic steps in Procedure LABEL:proc:lockstep. Let be the set returned by the procedure; we charge the symbolic steps in this call of the procedure to the vertices of the smaller set of and . By Theorem 1 we have either (a) , the number of symbolic steps in this call is bounded by , and the set is added to goodC or (b) and the number of symbolic steps in this call is bounded by . Case (a) can happen at most once for the vertices of , and for case (b) note that the size of a set containing a specific vertex can be halved at most times; thus we charge each vertex at most times. Hence we can bound the total number of symbolic steps in all calls to the procedure by . ∎

Appendix 0.C Details of Section 5: Symbolic MEC Decomposition

0.c.1 Basic Symbolic Algorithm for MEC decomposition

Recall that an end-component is a set of vertices that (a) has no random edges to vertices not in the set and its induced sub-MDP is (b) strongly connected and (c) contains at least one edge.

Algorithm LABEL:alg:mecbasic computes all maximal end-components of a given MDP and is formulated as to highlight the similarities to the algorithms for graphs and MDPs with Streett objectives. The algorithm maintains two sets, the set goodC of identified maximal end-components that is initially empty and the set of candidates for maximal end-components that is initialized with the SCCs of the MDP. In each iteration of the while-loop one set is removed from and either (1a) identified as a maximal end-component and added to goodC or (1b) removed because the induced sub-MDP does not contain an edge or (2) it contains vertices with outgoing random edges. In the latter case these vertices rout are identified and their random attractor is removed from . After this step the sub-MDP induced by the remaining vertices of might not be strongly connected any more. Therefore the SCCs of this sub-MDP are determined and added to as new candidates for maximal end-components. Note that this maintains the invariants that (i) each set in induces a strongly connected subgraph and (ii) each end-component is a subset of one set in either goodC or . By (i) a set in is an end-component if it does not have outgoing random edges and the induced sub-MDP contains an edge, i.e., in particular this holds for the sets added to goodC (soundness). By (ii) and at termination of the while-loop the algorithm identifies all maximal end-components of the MDP (completeness). Since both (1) and (2) can happen at most times, there are iterations of the while-loop. In each iteration the most expensive operations are the computation of a random attractor and of SCCs, which can both be done in symbolic steps. Thus Algorithm LABEL:alg:mecbasic correctly computes all maximal end-components of an MDP and takes symbolic steps.

algocf[t]    

0.c.2 Improved Symbolic Algorithm for MEC decomposition

Informal description. We show how to determine all maximal end-components (MECs) of an MDP in symbolic operations. The difference to the basic algorithm lies in the way strongly connected parts of the MDP are identified after the deletion of vertices that cannot be contained in a MEC. For this the symbolic lock-step search from Section 3 is used whenever not too many edges have been deleted since the last re-computation of SCCs.

Let be the given MDP and its underlying graph. The algorithm maintains two sets of vertex sets: the set goodC of already identified MECs that is initialized with the empty set and the set that is initialized with the SCCs of and contains vertex sets that are candidates for MECs. The algorithm preserves the following invariant for the goodC and over the iterations of the while-loop and returns the set goodC when the set is empty after an iteration of the while-loop.

Invariant 4 (Maintained Sets)

The sets in are pairwise disjoint and for every maximal end-component  of there exists a set such that either or .

algocf[htbp]    

For each vertex set  in additionally a subset of is maintained that contains vertices that have lost outgoing edges since the last time a superset of was identified as strongly connected. We use the following restrictions of Invariant 1 and Theorem 1 (presented in Section 3) to bottom SCCs only.

Invariant 5 (Start Vertices BSCC)

Either (a) is empty and is strongly connected or (b) at least one vertex of each bottom SCC of is contained in .

Theorem 5 (Lock-Step Search BSCC)

Provided Invariant 5 holds, Procedure LABEL:proc:lockstep(, , , ) returns a bottom SCC  of in symbolic steps.

Proof

The proof of Theorem 5 is a straightforward simplification of the proof of Theorem 1 located in Appendix 0.A. ∎

Initially the sets are empty. The algorithm maintains Invariant 5 for all . This will ensure the correctness and the number of symbolic steps of Procedure LABEL:proc:lockstep (Section 3) as called by the algorithm.

In each iteration of the while-loop one vertex set  is removed from and processed. First the random vertices of with edges to vertices of are identified and their random attractor is removed from . After this step, there are no random vertices with edges from to . The predecessors of the removed vertices that are contained in are added to and additionally is updated to only include vertices that are still in . This preserves Invariant 5 (see also [30, Lemma 4.5.2]). The number of symbolic steps for the attractor computation can be charged to the removed vertices and is therefore bounded by in total.

If afterwards does not contain an edge anymore, then is not considered further and the algorithm continues with the next iteration. Otherwise one of three cases happens.

Case (1): If is empty, then by Invariant 5 is strongly connected, contains at least one edge and does not contain a random vertex with edges to , i.e., is an end-component, and by Invariant 4 it is a MEC. In this case the algorithm adds the set to goodC, which preserves both invariants and can happen at most times.

Case (2): If there are at least vertices in , then the set is deleted and as in the basic algorithm all SCCs of are computed and add to as new candidates for MECs. For each of the SCCs  a set is initialized with the empty set. As a vertex is added to a set only if one of its incoming edges is removed by the algorithm, Case (2) can happen only times over the whole algorithm. Thus the total number of symbolic steps for this case is . Note that the Invariants 5 and 4 are preserved.

Case (3): If contains less than vertices, then Procedure LABEL:proc:lockstep(, , , ) is called. By Invariant 5 and Theorem 5 the procedure returns a bottom SCC  of in many symbolic steps. Since there are no random edges between and in and has no outgoing edges in , we have that is an end-component if it contains at least one edge. By Invariant 4 it is also a MEC and is correctly added to goodC. As the sets in goodC are not considered further by the algorithm, we can charge the symbolic steps of Procedure LABEL:proc:lockstep to the vertices of . Thus this part takes at most symbolic steps over the whole algorithm. The vertices of are added back to , which preserves Invariant 4. The predecessors of in are added to and vertices of are removed from , which preserves Invariant 5.

By the above case analysis we have that each vertex set that is added to goodC is indeed a MEC (soundness). By Invariant 4 and at termination of the algorithm we further have completeness. In each iteration either does not contain an edge and is not considered further, a set is added to goodC (and not contained in after that) or case (2) happens. Thus there are at most iterations of the algorithm. The symbolic operations we have not yet accounted for in the analysis of the number of symbolic steps are of per iteration. Hence Algorithm LABEL:alg:mecimpr takes symbolic steps and correctly computes the MECs of the given MDP .

Lemma 6 (Invariants of Improved Algorithm for MEC)

Invariant 5 and Invariant 4 are preserved throughout Algorithm LABEL:alg:mecimpr, i.e., they hold before the first iteration, after each iteration, and after termination of the main while-loop. Further, Invariant 5 is preserved during each iteration of the main while-loop.

Proof

Invariant 5. The proof of maintaining Invariant 5 in Algorithm LABEL:alg:mecimpr is a straightforward simplification of the proof of maintaining Invariant 1 in Algorithm LABEL:alg:streettgraphimpr (located in Appendix 0.B).

Invariant 4 – Disjointness. The sets in are pairwise disjoint at the initialization since goodC is initialized as . Furthermore, whenever a set is added to goodC in an iteration of the main while-loop, a superset is removed from in the same iteration of the while-loop. Therefore by induction the disjointness of the sets in is preserved.

Invariant 4 – Containment of maximal end-components. At initialization, contains all SCCs of . Each maximal end-component of is strongly connected, so there exists an SCC of such that .

Consider a set that is removed from at the beginning of an iteration of the main while-loop. Consider further a maximal end-component of such that . We require that a set is added to either or goodC in this iteration of the main while-loop.

First, after we remove from , we maintain the fact that by Lemma 2. Second, contains an edge since . Finally, one of the three cases happens:

Case (1): If , then the set is added to goodC.

Case (2): If , then the algorithm computes the SCCs of . Since is strongly connected, it is completely contained in some SCC of , and is added to .

Case (3): If , then the algorithm partitions into and . By Theorem 5 we have that is a (bottom) SCC of . Since is strongly connected, it is completely contained in some SCC of . Therefore either or . The set is added to . If , then in particular contains an edge, and is added to goodC.

By the above case analysis we have that a set is added to either or goodC in the iteration of the main while-loop. ∎

Proof (of Theorem 3)

Correctness. A candidate set can be added to goodC in three cases. When is added to goodC at line LABEL:l:mec:good1 (resp. at line LABEL:l:mec:good2), then it contains an edge by the check at line LABEL:l:mec:p1, it is strongly connected by and Invariant 5 (resp. by the result of allSCCs), and it has no random vertices with edges to by the random attractor removal at lines LABEL:l:mec:remb-LABEL:l:mec:reme. When is added at line LABEL:l:mec:good3, then it contains an edge by the check at line LABEL:l:mec:p2, it is strongly connected by Theorem 5, it contains no random vertices with edges to by the random attractor removal at lines LABEL:l:mec:remb-LABEL:l:mec:reme, and it contains no random vertices with edges to by the fact that is a bottom SCC of (see Theorem 5). Therefore we have that whenever a candidate set is added to goodC, it is an end-component, and by induction and Invariant 4 we have that it is a maximal end-component (soundness).

Finally, by soundness, Invariant 4, the termination of the algorithm (shown below), and the fact that at the termination of the algorithm, we have that goodC contains all the maximal end-components of (completeness).

Symbolic steps analysis. By [24], the initialization with the SCCs of a given MDP takes symbolic steps.

In each iteration of the outer while-loop, a set  is removed from and (a) is added to goodC, or (b) at least two sets that are (subsets of) a partition of are added to , or (c) is partitioned into two sets, one of them may be added to goodC and the other is added to . All three cases can happen at most times, so there can be at most iterations of the outer while-loop. The and operations at lines LABEL:l:mec:rembLABEL:l:mec:remeLABEL:l:mec:p1LABEL:l:mec:p2, and LABEL:l:mec:p3 can be charged to the iterations of the outer while-loop.

Each operation executed as a part of the random attractor computation at line LABEL:l:mecimpr:remattr adds at least one vertex to , and the vertices of are then not considered any further in the algorithm. Therefore there can, in total, be at most operations over all attractor computations at line LABEL:l:mecimpr:remattr.

Note that every vertex in each of can be attributed to at least one unique implicit edge deletion since we only add vertices to that are predecessors of the vertices that were separated from (or deleted from the maintained graph). Whenever the case occurs, for all subsets that are then added to , we initialize . Therefore, the case can happen at most times throughout the algorithm since there are at most edges that can be deleted. By [24] we have a bound for one iteration, so we can bound the total number of symbolic steps in all iterations of this case by .

It remains to bound the number of symbolic steps in Procedure LABEL:proc:lockstep. Let be the set returned by LABEL:proc:lockstep(, , , ). By Theorem 5 and the fact that , the number of symbolic steps in this call is bounded by , and the set is not considered further in the algorithm after this call. Hence we can bound the total number of symbolic steps in all calls of the procedure by . ∎

Appendix 0.D Details of Section 6: MDPs with Streett Objectives

0.d.1 Basic Symbolic Algorithm for MDPs with Streett Objectives

algocf[htbp]    

The pseudocode of the basic symbolic algorithm for MDPs with Streett objectives is given in Algorithm LABEL:alg:streettmdpbasic. The key differences compared to Algorithm LABEL:alg:streettgraphbasic are as follows: (a) SCC computation is replaced by MEC computation; (b) along with the removal of bad vertices, their random attractor is also removed; and (c) removing the attractor ensures that the check required for trivial SCCs for graphs (line LABEL:l:gbasic:edge) is not required any further.

To compute the almost-sure winning set for MDPs with Streett objectives, we first find all (maximal) good end-components and then solve almost-sure reachability with the union of the good end-components as target set as the last step of the algorithm. This is correct by Lemma 1. Towards finding all good end-components, the algorithm maintains two sets, the set goodEC of identified good end-components that is initially empty and the set of end-components that are candidates for good end-components that is initialized with the MECs of the MDP. In each iteration of the while-loop one set is removed from the set of candidates and the set of bad vertices of is determined. If is empty, then is a good end-component and added to goodEC. Otherwise the random attractor of in is removed from , which by Corollary 1 does not remove any vertices that are in a good end-component. The remaining vertices of have no outgoing random edges and thus still induce a sub-MDP but the sub-MDP might not be strongly connected any more. Then the MECs of this sub-MDP are added to . These operations maintain the invariants that (i) each set in is an end-component and (ii) each good end-component is a subset of one set in either goodEC or . By (i) a set in is a (maximal) good end-component if it does not contain any bad vertices, i.e., in particular this holds for the sets added to goodEC (soundness). By (ii) and at termination of the while-loop the algorithm identifies all (maximal) good end-components of the MDP (completeness). Since in each iteration of the while-loop either (1) a set is removed from and added to goodEC or (2) bad vertices are removed from a set and not considered further by the algorithm, there can be at most iterations of the while-loop. Furthermore, whenever bad vertices are removed, then the number of target pairs a given candidate set intersects is reduced by one. Thus each vertex is considered in at most iterations of the while-loop. The most expensive operation in the while-loop is the computation of the MECs. Denoting the number of symbolic steps for the MEC computation with , the number of symbolic steps of Algorithm LABEL:alg:streettmdpbasic is (assuming that the number of symbolic steps for the almost-sure reachability computation is lower than that).

0.d.2 Improved Symbolic Algorithm for MDPs with Streett Objectives

We present the technical details regarding the improved symbolic algorithm for MDPs with Streett objectives. The main ideas of the algorithm are presented in Section 6. The pseudocode is given in Algorithm LABEL:alg:streettmdpimpr.

algocf[htbp]    

The following invariant is maintained throughout Algorithm LABEL:alg:streettmdpimpr for the sets in goodEC and .

Invariant 7 (Maintained Sets)

The sets in are pairwise disjoint and for every good end-component  of there exists a set such that either or .

Furthermore, the algorithm maintains the invariant that each candidate for a good end-component contains no random edges to vertices not in .

Invariant 8 (No Random Outgoing Edges)

Given an MDP  and its underlying graph , for each set there are no random vertices in with edges to vertices in .

Finally, for each candidate set the algorithm remembers sets and of vertices that have lost incoming resp. outgoing edges since the last time a superset of was identified as being strongly connected. The algorithm maintains Invariant 1 and therefore it can use Procedure LABEL:proc:lockstep together with its correctness guarantee and bound on symbolic steps provided by Theorem 1.

Lemma 9 (Invariants of Improved Algorithm for MDPs)

Invariant 1, Invariant 7, and Invariant 8 are preserved throughout Algorithm LABEL:alg:streettmdpimpr, i.e., they hold before the first iteration, after each iteration, and after termination of the main while-loop. Further, Invariant 1 is preserved during each iteration of the main while-loop.

Proof

Invariant 1. The proof is a minor extension of the maintenance proof for Algorithm LABEL:alg:streettgraphimpr that is given in Appendix 0.B. In terms of strong connectivity of a candidate and the maintenance of the sets and , the only difference to the graph case is that after an SCC  is computed by allSCCs or Procedure LABEL:proc:lockstep, another subset of vertices  (vertices with outgoing random edges and their random attractor) is removed from . In this case the invariant is maintained by initializing resp.  with the vertices of with edges from resp. to vertices of , i.e., and .

Invariant 7 – Disjointness. The sets in are pairwise disjoint at the initialization since goodEC is initialized as . Furthermore, whenever a set is added to goodEC in an iteration of the main while-loop, a superset is removed from in the same iteration of the while-loop. Therefore by induction the disjointness of the sets in is preserved.

Invariant 7 – Containment of good end-components. At initialization, contains all MECs of the input MDP . Each good end-component of