1 Introduction
Computing reachability probabilities is a fundamental problem in the analysis of probabilistic systems. It is known that probabilistic model checking titleems can be solved via reachability probabilities [4], much like nondeterministic model checking problems are reduced to emptiness and hence to reachability [34]
. While the computation of reachability probabilities for finitestate systems is effectively solved by linear programming, the problem becomes much more challenging for
probabilistic programs—a paradigm that attracts growing attention as a programming language foundation for machine learning
[17]—because their transition graphs are infinite in general.Reachability probabilities of probabilistic programs with while loops are clearly not computable, because the problem encompasses termination of (nonprobabilistic) while programs. Therefore the existing research efforts have focused on sound approximation methods for reachability probabilities. An approach that is widely used in the literature is to use ranking supermartingales—a probabilistic analogue of ranking functions—as a witness for the qualitative question of almostsure reachability. Ranking supermartingales are amenable to templatebased synthesis [8, 11, 9], making them appealing from the automatic analysis point of view. Recently, methods for quantitatively underapproximating reachability probabilities are also proposed in [13, 33].
The dual question of overapproximating reachability probabilities, which can then be used to qualitatively refute almostsure reachability, is also considered. In the control theory, supermartingales are used as a probabilistic counterpart of barrier certificates [26, 31]. A similar idea is recently used for the purpose of synthesizing stochastic invariants for probabilistic programs [13]. Here an overapproximation of reachability probability serves as quantitative verification for safety: it gives an upper bound for the probability that the system or the program reaches a bad state.
certificate for  from  

ranking (super and sub) martingale for underapproximation  additive supermartingale (ARnkSupM, §5)  [24, 8]  
scaled submartingale (SclSubM, §6)  this paper for PP, following categorical observations in [33] for MC  
repulsing supermartingale for overapproximation  decreasing supermartingale (RepSupM, §3)  [13], derived from Azuma’s martingale concentration inequality  
nonnegative supermartingale (NNRepSupM, §4)  this paper, derived from the Knaster–Tarski theorem ([26, 31], without nondeterminism, derived from Markov’s concentration inequality) 
Martingalebased techniques for approximation of reachability probabilities. MC stands for Markov chains, and PP stands for probabilistic programs
Table 1 lists four supermartingalebased techniques for over and underapproximating reachability probabilities. The table is not meant to be exhaustive—still, it shows that multiple methods have been introduced and studied, in different communities (formal verification, control theory, etc.) and with different mathematical backgrounds (ranking functions, martingale concentration inequalities, etc.).
The current work aims at a comprehensive and comparative account of those martingalebased techniques in Table 1. Central to our account is the role of ordertheoretic fixed points, a classic topic in theoretical computer science. More specifically, we characterize our objectives—namely reachability probability and expected reaching time—as suitable least fixed points. It turns out that a large part of the theory of martingalebased methods can be developed based on this ordertheoretic characterization, without using mathematical gadgets unique to probabilistic settings such as martingale concentration inequalities. Our contributions are summarized as follows.

A comprehensive and comparative account of different martingalebased techniques for approximating reachability probabilities. We identify their key mathematical principles to be ordertheoretic fixed points and martingale concentration inequalities, and we emphasize the role of the former.

We introduce two martingalebased techniques that seem to be new, namely SclSubM and NNRepSupM in Table 1. Their purely probabilistic versions have been in the literature: SclSubM is from a categorytheoretic account in [33], and NNRepSupM is from control theory [31]. We extend them to probabilistic programs that additionally have nondeterminism. Moreover, completeness of ARnkSupM for probabilistic programs with realvalued variables seems to be new.

We formalize those techniques, taking probabilistic programs (with nondeterminism) as the target of analyses. We investigate soundness and completeness of the techniques in Table 1. While the ordertheoretic fixedpoint foundation gives us clear theoretical guidance, additional nondeterminism requires us to carefully establish measuretheoretic arguments.
The paper is organized as follows. Preliminaries are in §2, where we introduce our system models (pCFGs) for operational semantics of probabilistic programs, and review the theory of ordertheoretic fixed points (the Knaster–Tarski and Cousot–Cousot theorems). In §3–6 we discuss the four techniques in Table 1, offering a unifying account based on ordertheoretic fixed points, and providing some new techniques and results. In §7 we give implementations and experiment results of templatebased synthesis. After discussion of related work in §8, we conclude in §9.
2 Preliminaries
We first fix some notations. We write and for the set of all natural numbers (i.e. nonnegative integers) and reals, respectively. We use subscripts to denote subsets of and ; for example, denotes the set of all nonnegative reals. We write for the sets of all finite, nonempty finite, and infinite sequences of elements of , respectively.
We use the Borel measurable structure of the set of real numbers. This induces the measurable structures of all the other sets used in this paper: where , where is finite, and so on. The induced measurable structures are defined in a standard manner: for example, where is finite, it is given by
. The set of probability distributions on
is denoted by . The Dirac measure on is denoted by . The support of is defined by . The set of all Borel measurable function from to is denoted by . The functions and are the realvalued constant function of which coefficient is and , respectively.2.1 Probabilistic Control Flow Graphs (pCFGs)
We take the notion of pCFG from [2]
and use it as our model of probabilistic systems. pCFGs can be thought of as a subclass of Markov decision processes (MDPs), but tailored for operational semantics of probabilistic programs (§
2.2).Definition 2.1 (pCFG, [2])
A probabilistic control flow graph (pCFG) is a tuple consisting of the following components.

A finite set of locations, equipped with a partition into nondeterministic, probabilistic, deterministic and assignment locations.

A finite set of program variables.

A transition relation which is total (each location has a successor). For , we write to denote the set of all successors of , i.e. . We require that each assignment location has a unique successor; in this case, denotes this unique location.

An update function , where . Here, three components of represent deterministic, probabilistic and nondeterministic assignment, respectively.

A family of probability distributions.

A guard function such that, for each , the following hold: (collective exhaustion) ; and (mutual exclusion) , and imply . We write if .
A configuration of a pCFG is a pair of a location and a vector. A successor of a configuration is a one such that and
A finite path of is a finite sequence of configurations where is a successor of for each . Similarly, A run of is an infinite sequence of configurations such that each is a successor of .
Schedulers resolve nondeterminism. Given a history of configurations, it gives a distribution of the successor’s location or valuation vector. We assume that a scheduler is universally measurable, which is standard in control theory (see e.g. [5]).
If a pCFG and a scheduler for are given, then the behavior of is determined for each initial configuration ; we represent it by the map . For each nonempty sequence the distribution is, intuitively, the distribution of the next configuration given a current history of configurations under the scheduler . For the set of all schedulers for we define the following.
Definition 2.2 (reachability probabilities )
Let be a pCFG. The reachability probability from a configuration to a region under a scheduler is defined by
for the case of , and otherwise. The upper reachability probability from to is defined by ; the lower reachability probability is defined by .
Definition 2.3 (reaching times )
Let be a pCFG. The expected reaching time of from a configuration to under a scheduler is defined by for ; for it is defined by
if , or otherwise. The upper expected reaching time of from to is , and the lower expected reaching time is .
2.2 Probabilistic Programs: APP and PPP
The goal of this paper is the reachability analysis of imperative programs with probabilistic and nondeterministic branching. We consider two languages taken from [13, 11], called affine probabilistic programs (APP) and polynomial probabilistic programs (PPP). The two languages differ only in the arithmetic expressions allowed in the assignment commands and Boolean expressions. For example, the assignment command is allowed in PPP but not in APP; is allowed in both since its righthand side is an affine expression.
Both APP and PPP have the standard control structure in imperative languages—such as ifbranches and whileloops. APP and PPP additionally have nondeterministic and probabilistic ifbranches (if then and if then , respectively, where ). They also have nondeterministic and probabilistic assignment commands: where a value is chosen from a set ; and where a value is sampled from a probability distribution over .
The definition of the semantical model pCFG (§2.1) mirrors the structure of these languages. The translation from APP/PPP to pCFGs is straightforward and omitted.
2.3 OrderTheoretic Foundation of Fixed Points
Ordertheoretic fixed points are central to computer science, for recursive computation, inductive/coinductive datatypes and reasoning and specification of reactive behaviors, etc. In general, a fixedpoint equation can have multiple solutions; often we are interested in extremal solutions: least fixed points (lfp’s, for liveness, induction, etc.) and greatest ones (gfp’s, for safety, coinduction, etc.). The following fundamental results (in a simple setting of complete lattices) give two different characterizations of lfp’s and gfp’s.
Theorem 2.4
Let be a complete lattice, and be a monotone function. Then has the least fixed point and the greatest . Moreover,

(Knaster–Tarski) The lfp is the least prefixed point: . Similarly, the gfp is the greatest postfixed point: .

(Cousot–Cousot [14]) The (potentially transfinite) ascending chain stabilizes to . Here is defined by obvious induction: for a successor ordinal; and for a limit ordinal.
Similarly, the descending chain stabilizes to . ∎
From these characterizations we can derive the following reasoning principles.
Corollary 2.5
(lfpKT) implies . (gfpKT) implies . (lfpCC) For each ordinal , . (gfpCC) For each ordinal , . ∎The arguments so far are symmetric for lfp’s and gfp’s. However, if one turns to the common proof methods for lfp specifications (termination, reachability, liveness) and those for gfp specifications (safety), a strong contrast emerges. Here is an example.
Lemma 2.6
Let be a Kripke frame, and . (Invariant for safety) Let be an invariant, that is, . Here is defined by . Assume also that . Then implies that there is no path from to . (Ranking function for liveness) Let be a ranking function for . That is, 1) for each , there is a successor such that ; and 2) for each , implies . Then, implies that there is a path from to . ∎Knaster–Tarski  Cousot–Cousot  

lfp  overapprox.  underapprox. 
gfp  underapprox.  overapprox. 
The difference between the two methods is accounted for by the fact that, in Cor. 2.5, two items give underapproximations while the other two give overapproximations. It is clear that the invariant method in Lem. 2.6 comes from (gfpKT) of Cor. 2.5. Its dual, (lfpKT), gives only an overapproximation —it can be used for refutation but not for verification. Similarly, ranking functions come from (lfpCC)—the role of wellfoundedness of the value domain mirrors the structure of ordinals. Its dual (gfpCC) only gives an overapproximation of . The situation is summarized in the above table.
The above foundations underpin our technical developments: this is because reachability probabilities and reaching times are characterized as least fixed points. We note that our semantical domains in later sections need not be complete lattices. In those cases we exploit the  and cpo structures, the corresponding continuity of , and the Kleene theorem. The last is understood as a variation of the Cousot–Cousot theorem.
2.4 Invariants and the Nexttime Operations
Definition 2.7 ((pure) invariant for pCFG)
Let be a pCFG. A measurable set is called a (pure) invariant for if , and for each , if is a successor of then .Definition 2.8 (the “nexttime” operation )
Let be a pCFG, be a pure invariant and . For a measurable we define the function of the same type as as follows, provided the righthand side of each equation is welldefined.

For , .

For , .

For , where is the unique location s.t. .

For , let .

if is a measurable function.

if is a distribution.

if is a measurable set.

The function is defined as above, but replacing with in the first line and with in the last.
Proposition 2.9
We define a pointwise partial order on , i.e. if and only if holds for every . Let be a proper closed convex subset of . Then and are welldefined for every , and the following hold. The operators and are monotone endofunctions over . In particular, and are Borel measurable for any . is continuous, and is continuous. ∎3 Decreasing Repulsing Supermartingales (RepSupM)
In §3–6 we will discuss the four martingalebased techniques in Table 1. Here we briefly review the notion of decreasing repulsing supermartingale (RepSupM) from [13]. It is, to the best of our knowledge, the only existing martingalebased notion for overapproximating reachability probabilities.
Definition 3.1 (RepSupM [13])
Let be a pCFG, be a pure invariant, and be a Borel set. An repulsing supermartingale (RepSupM) for supported by is a measurable function such that i) for each , and ii) for each .Theorem 3.2 (soundness, [13])
Suppose there exists an RepSupM for supported by such that . Further assume that has bounded differences for some , i.e. for each and its successor it holds . Let and . We have the following inequality:We note that for any that has bounded differences, the function is welldefined. The bound in (1) is derived from Azuma’s concentration inequality, a wellknown martingale concentration lemma that exploits bounded differences. RepSupM is not complete: there exist a pCFG and a set of configurations such that but no RepSupM can prove it. See Fig. 3 below.
4 Nonnegative Repulsing Supermartingales (NNRepSupM)
We move on to another notion for overapproximating reachability probabilities, nonnegative repulsing supermartingale (NNRepSupM). We believe this is new. Compared to the notion of RepSupM, NNRepSupM has the following features.

NNRepSupM is derived from the theory of ordertheoretic fixed points (§2.3), unlike RepSupM that relies on Azuma’s martingale concentration lemma.

Consequently, we can show soundness and completeness of NNRepSupM rather easily, while RepSupM is sound but not complete.

We experimentally observe that NNRepSupM often gives better bounds (§7).
The definition of NNRepSupM resembles probabilistic barrier certificates used in control theory [26, 31]. Our technical contributions are the following: i) we develop the theory of NNRepSupM in the presence of nondeterminism, while the settings in [26, 31] are purely probabilistic; and ii) we characterize NNRepSupM in the general terms of ordertheoretic fixed points (§2.3), unlike the previous theory in [26, 31] that relies on Markov’s martingale concentration lemma.^{1}^{1}1We note that the theory of NNRepSupM can also be developed using Markov’s lemma. The latter unveils the mathematical similarity between NNRepSupM and ARnkSupM (§5).
The notion comes with upper and lower variants. They are used to overapproximate and , respectively (Def. 2.2). In this section we use .
Definition 4.1 (NNRepSupM for pCFG)
Let be a pCFG, be a pure invariant, and be a Borel set. An upper nonnegative repulsing supermartingale (UNNRepSupM) over for supported by is a function s.t.
The function is a lower nonnegative repulsing supermartingale (LNNRepSupM) if it satisfies the above conditions, but with replaced with .
We shall prove soundness and completeness of NNRepSupM, based on the foundations in §2.3. The following characterization is fundamental.
Proposition 4.2
In the setting of Def. 4.1, we define endofunctions and over as follows:
Then the upper reachability probability ^{2}^{2}2 Precisely it is the restriction of to ; in what follows we do this identification for , , , and . is the least fixed point (lfp) of . Similarly, is the lfp of .
Proof
(Sketch) We first need to show that and are Borel measurable. This is not very easy, as they are defined via supremum or infimum over uncountably many schedulers. We use the technique of optimal scheduler known from control theory [5].
Checking that and are fixed points is not hard, though laborious. We use optimal schedulers again for interchange between sup./inf. and integration.
Finally, the proofs for minimality differ for and . For , we first observe that is continuous (immediate from Prop. 2.9). Therefore by the Kleene theorem, the lfp of is given by (i.e. the chain in Thm. 2.4 stabilizes after steps). We can check the coincidence between and by direct calculation.
For , let be a fixed point of . Then for each , we can construct a scheduler such that for each (at the th step, chooses a optimal successor). Since , this proves . ∎
It is easy to see that a UNNRepSupM is nothing but a prefixed point of (i.e. ), and that an LNNRepSupM is a prefixed point of . Therefore, soundness and completeness of NNRepSupM follow essentially from Cor. 2.5.
Corollary 4.3

(Soundness) If is a UNNRepSupM for supported by , then for each we have .
Similarly, if is an LNNRepSupM for supported by , then for each we have . This means, concretely, that for each there is a scheduler such that, for any , we have .

(Completeness) There exists a UNNRepSupM that gives the optimal bound for . The same for LNNRepSupM. ∎
5 Additive Ranking Supermartingales (ARnkSupM)
We move on to the notion of additive ranking supermartingale (ARnkSupM) in Table 1. It is the bestknown martingalebased notion for analysis of probabilistic programs and is used for overapproximating the expected reaching time. That its value is finite implies almostsure reachability, too. We review its theory; the reason is to demonstrate that the same ordertheoretic structure (see §2.3) underlies ARnkSupM and NNRepSupM in the previous section. The completeness result ((2) of Cor. 5.3) for pCFGs with realvalued variables seems new, too; See §8 for a detailed comparison to existing works. Proofs are done in a much similar manner to the ones in §4. In this section we use .
We note that completeness of UARnkSupM we state below is the one for strong almostsure reachability [3]. UARnkSupM is incomplete for positive almostsure reachability [15], that is, it cannot witness the condition in general.
Definition 5.1 (ARnkSupM for pCFG, [8])
Let be a pCFG, be a pure invariant, and be a Borel set. An upper additive ranking supermartingale (UARnkSupM) over for supported by is a function that satisfies for each .
The function is a lower additive ranking supermartingale (LARnkSupM) if it satisfies the above conditions, but with replaced with .
Proposition 5.2
In the setting of Def. 5.1, we define endofunctions and over as follows:
Then the upper expected reaching time is the lfp of . Similarly, is the lfp of . ∎
Corollary 5.3

(Soundness, e.g. [2]) If is a UARnkSupM for supported by , then for each we have . In particular, for each that satisfies we have .
Similarly, if is an LARnkSupM for supported by , then for each we have . This means, concretely, that for each there is a scheduler such that, for any , we have . In particular, for each that satisfies we have .

(Completeness) There exists a UARnkSupM that gives the optimal bound for . The same holds for LARnkSupM. ∎
6 Scaled Submartingales (SclSubM)
Here we present the theory of scaled submartingales (SclSubM). It is for underapproximating reachability (Table 1). Compared to the wellknown method of ARnkSupM, the greatest advantage is in quantitative reasoning: the value of a SclSubM is guaranteed to be below the reachability probability (which can be less than ), while ARnkSupM is useful only if almost reachability holds. In this section we use .
The notion of SclSubM is first introduced in [33], as an instance of a categorical abstraction of ranking functions. The current paper’s contribution lies in the following: i) the theoretical developments about SclSubM in concrete (noncategorical) terms; ii) introduction of nondeterminism (the setting of [33] is purely probabilistic); and iii) templatebased synthesis of SclSubM.
Definition 6.1 (SclSubM for pCFG, [33])
Let be given. An upper Scaled Submartingale (USclSubM) over for supported by is a function that satisfies for each . A lower Scaled Submartingale (LSclSubM) over for supported by is a function that satisfies for each .The derivation of SclSubM, from a categorical account in [33], can be described in the following concrete terms. A SclSubM is a postfixed point of certain functions (namely and below). According to (gfpKT) in Cor. 2.5, SclSubM underapproximates a greatest fixed point—but reachability is a least fixed point. The trick here is as follows: 1) thanks to the scaling by , the gfp and lfp of coincide; and 2) the lfp (hence the gfp) of is easily seen to be below the lfp of , that is, the reachability probability that we are after. The overall argument signifies the role of the Knaster–Tarski theorem.
Proposition 6.2
Let and be as defined in Prop. 4.2. Define endofunctions and over as follows: and Then we have i) and , and ii) and . ∎
Corollary 6.3 (soundness)
If is a USclSubM for supported by , then for each we have . This means, concretely, that for each there is a scheduler such that, for any , we have .
Similarly, if is an LSclSubM for supported by , then for each we have .
Proof
Just notice that if is an upper or lowerSclSubM, then so is . The rest is as described in the paragraph before Prop. 6.2. ∎
7 Implementation and Experiments
We implemented templatebased automated synthesis algorithms for NNRepSupM (§4) and SclSubM (§6), and present some experimental results. We implemented the following programs:

synthesis of a UNNRepSupM for an APP based on a linear template.

synthesis of a UNNRepSupM for a PPP based on a polynomial template.

synthesis of an LSclSubM for an APP based on a linear template.
Each algorithm first translates given an APP or a PPP to a pCFG and a terminal configuration , and then solves an optimization problem of finding a UNNRepSupM (LSclSubM) over for that gives a small (large) value as possible at the initial configuration. Reduction of optimization problems to LP or SDP ones are done in standard ways in the literature; we use Farkas’ lemma (see e.g. [8, 13]) for the case of APPs, and Schmüdgen’s Positivstellensatz (see e.g. [11, 9]) for PPPs.
We have augmented the syntax of APPs and PPPs (§2.2) so that we can specify an invariant and a terminal configuration . The program does not synthesize an invariant nor prove the correctness of the given invariant, and therefore the user has to provide a correct invariant by hand or by using some algorithm, e.g. [22].
All the programs are implemented in OCaml. We have used glpk (v4.63) [16] and SDPT3 [29] for the LP and SDP solvers respectively. For the implementation of Prog. II, we have also made use of a MATLAB toolbox SOSTOOLS (v3.03) [30].
We tested our implementations for several APPs and PPPs. We have used different benchmark sets for Prog. I–II and Prog. III because what is overapproximated by Prog. I–II () and what is underapproximated by Prog. III () are different. The benchmarks implement the following probabilistic processes that are used as benchmarks in the literature. More details and codes are given in §0.G.

(Adversarial random walk) A variation of a random walk, whose analysis is more challenging because of additional adversarial nondeterministic choices [12]. We have considered three variants: (a1) 1D, (a2) 2D and (a3) a variant of 2D. (a1) is a random walk over modeling a discrete queuing system, and is parametrized by that determines the distribution of the number of packets that arrive in each round. (a2) and (a3) are random walks over parametrized by . They determine the distribution of movement distances in each round. We added a queue size limit for (a1) and a time limit for (a2) and (a3). If the queue size exceeds in (a1) or rounds were consumed in (a2) or (a3), the program stops, and it is not counted as termination.
We have coded (a)–(b) as an APP. Experiments for Prog. I and III were carried out on a MacBook Pro laptop with a Core i5 processor (2.6 GHz, 2 cores) and 16 GiB RAM. That for Prog. II was carried out on an Amazon EC2 c4.large instance (May 2018, 2 vCPUs and 3.75 GiB RAM) running Ubuntu 16.04.4 LTS (64 bit). The results are in Table. 3–3. For each program, the first column (“time (s)”) shows the total execution time, and the second column (“bound”) shows the calculated probability bound.
(Applicability of NNRepSupM) Table 3 shows the results for Prog. I–II; the goal of these experiments is to certify the applicability of NNRepSupM to programs with nondeterminism (a1). We have tested them for (a1) with two combinations of parameters. Prog. I–II found a nontrivial bound for the reachability probability when while it failed to find such a bound when . Intuitively, the random walk is more “unfavorable” in the former case in the sense that the opposite direction from a terminal configuration is chosen in higher probabilities. As expected, a polynomial NNRepSupM gives tighter bound than a linear one, but it took much longer. The bound was not improved by increasing the degree of the polynomial template.
(Applicability of SclSubM) Table 3 shows the results for Prog. III; here we wish to certify applicability of our new method SclSubM. For each variant of (a), we have tested Prog. III for two combinations of parameters. In each variant, Prog. III gives a nontrivial probability bound for one combination and a trivial bound for the other combination. In fact, all the cases where nontrivial bounds were “favorable” random walks where the direction to a terminal configuration tends to be chosen. In contrast, the cases where no nontrivial bound was found were “unfavorable” random walks. Note that this is the converse of the results for Prog. I–III. Prog. III also succeeded in giving a nontrivial bound for (b). However, if we increase the parameter (i.e. if we strengthened the power of air conditioners), it failed to give a nontrivial bound.
Comments
There are no comments yet.