1 Introduction
Stochastic model checking is a formal method that designers and engineers can use to determine the likelihood of safety and liveness
properties. Checking properties using numerical model checking techniques requires enumerating the state space of the system to determine the probability that the system is in any given state at a desired time
[18]. Realworld applications often have very large or even infinite state spaces.Numerous state representation, reduction, and approximation methods have been proposed. Symbolic model checking based on multiterminal binary decision diagrams (MTBDDs) [24] has achieved success in representing large Markov Decision Process (MDP) models with a few distinct probabilistic choices at each state, e.g., the shared coin protocol [3]
. MTBDDs, however, are often inefficient for models with many different and distinct probability/rate values due to the inefficient representation of solution vectors.
Continuoustime Markov chain (CTMC) models, whose state transition rate is a function of state variables, generally contain many distinct rate values. As a result, symbolic model checkers can run out of memory while verifying a typical CTMC model with as few as 73,000 states [24]. State reduction techniques, such as bisimulation minimization [15, 7, 8], abstraction [21, 15, 6, 13], symmetry reduction [17, 5], and partial order reduction [9] have been mainly extended to discretetime, finitestate probabilistic systems. The threevalued abstraction [15] can reduce large, finitestate CTMCs. It may, however, provide inconclusive verification results due to abstraction.To the best of our knowledge, only a few tools can analyze infinitestate probabilistic models, namely, STAR [20] and INFAMY [10]. The STAR tool primarily analyzes biochemical reaction networks. It approximates solutions to the chemical master equation (CME) using the method of conditional moments (MCM) [12]
that combines momentbased and statebased representations of probability distributions. This hybrid approach represents species with low concentrations using a discrete stochastic description and numerically integrates a small master equation using the fourth order RungeKutta method over a small time interval
[2]; and solves a system of conditional moment equations for higher concentration species, conditioned on the low concentration species. This method has been optimized to drop unlikely states and add likely states onthefly. STAR relies on a wellstructured underlying Markov process with small sensitivity on the transient distribution. Also, it mainly reports state reachability probabilities, instead of checking a given probabilistic property. INFAMY is a truncationbased approach that explores the model’s state space up to a certain finite depth . The truncated state space still grows exponentially with respect to exploration depth. Starting from the initial state, breadthfirst state search is performed up to a certain finite depth. The error probability computed during the model checking depends on the depth of state exploration. Therefore, higher exploration depth generally incurs lower error probability.This paper presents a new infinitestate stochastic model checker, STochastic Approximate Modelchecker for INfinitestate Analysis
(STAMINA). Our tool also takes a truncationbased approach. In particular, it maintains a probability estimate of each path being explored in the state space, and when the currently explored path probability drops below a specified threshold, it halts exploration of this path. All transitions exiting this state are redirected to an absorbing state. After all paths have been explored or truncated, transient Markov chain analysis is applied to determine the probability of a transient property of interest specified using
Continuous Stochastic Logic (CSL) [4]. The calculated probability forms a lower bound on the probability, while the upper bound also includes the probability of the absorbing state. The actual probability of the CSL property is guaranteed to be within this range. An initial version of our tool and preliminary results are reported in [23]. Since that paper, our tool has been tightly integrated within the PRISM model checker [19] to improve performance, and we have also developed a new propertyguided state expansion technique to expand the state space to tighten the reported probability range incrementally. This paper reports our results, which show significant improvement on both efficiency and verification accuracy over several nontrivial case studies from various application domains.2 Stamina
Figure 1 presents the architecture of STAMINA. Based on a userspecified probability threshold (kappa), it first constructs a finitestate CTMC model from the original infinitestate CTMC model using the state space approximation method presented in Section 2.1. is then checked using the PRISM explicitstate model checker against a given CSL property , where and (for cases where it is desired that a predicate be true within a certain probability bound) or (for cases where it is desired that the exact probability of the predicate being true be calculated). Lower and upperbound probabilities that holds, namely, and , are then obtained, and their difference, i.e., , is the probability accumulated in the absorbing state which abstracts all the states not included in the current state space. If , it is not known whether holds. If exact probability is of interest and the probability range is larger than the userdefined precision , i.e., , then the method does not give a meaningful result.
For an inconclusive verification result from the previous step, STAMINA applies a propertyguided approach, described in Section 2.2, to further expand , provided is a nonnested “until” formula; otherwise, it uses the previous method to expand the state space. Note that also drops by the reduction factor to enable states that were previously ignored due to a low probability estimate to be included in the current state expansion. The expanded CTMC model is then checked to obtain a new probability bound . This iterative process repeats until one of the following conditions holds: (1) the target probability falls outside the probability bound , (2) the probability bound is sufficiently small, i.e, , or (3) a maximal number of iterations has been reached ().
2.1 State Space Approximation
The state space approximation method [23] truncates the state space based on a userspecified reachability threshold . During state exploration, the reachabilityvalue function, , estimates the probability of reaching a state onthefly, and is compared against to determine whether the state search should terminate. Only states with a higher reachabilityvalue than the reachability threshold are explored further.
Figure 2 illustrates the standard breadth first search (BFS) state exploration for reachability threshold . It starts from the initial state whose reachabilityvalue i.e., , is initialized to as shown in Figure 1(a). In the first step, two new states and are generated and their reachabilityvalues are and , respectively, as shown in Figure 1(b). The reachabilityvalue in is distributed to its successor states, based on the probability of outgoing transitions from to its successor state. For the next step, only state is scheduled for exploration because . Note that the transition from to is executed because is already in the explored set. Expanding leads to two new states, namely and as shown in Figure 1(c), from which only is scheduled for further exploration. This leads to the generation of and shown in Figure 1(d). State exploration terminates after Figure 1(e) since both newly generated states have reachabilityvalues less than . States , , and are marked as terminal states. During state exploration, the reachabilityvalue update is performed every time a new incoming path is added to a state because a new incoming path can add its contribution to the state, potentially bringing the reachabilityvalue above , which in turn changes a terminal state to be nonterminal. When the truncated CTMC model is analyzed, it introduces some error in the probability value of the property under verification, because of leakage the probability (i.e., cumulative path probabilities of reaching states not included in the explored state space) during the CTMC analysis. To account for probability loss, an abstract absorbing state is created as the sole successor state for all terminal states on each truncated path. Figure 1(e) shows the addition of the absorbing state.
2.2 Property Based State Space Exploration
This paper introduces a propertyguided state expansion method, in order to efficiently obtain a tightened probability bound. Since all nonnested CSL path formulas (except those containing the “next” operator) derive from the “until” formula, , construction of the set of terminal states for further expansion boils down to eliminating states that are known to satisfy or dissatisfy . Given a state graph, a path starting from the initial state can never satisfy , if it includes a state satisfying . Also, if a path includes a state satisfying , satisfiability of can be determined without further expanding this path beyond the first state. Our propertyguided state space expansion method identifies the path prefixes, from which satisfiability of can be determined, and shortens them by making the last state of each prefix absorbing based on the satisfiability of . Only the nonabsorbing states whose path probability is greater than the state probability estimate threshold are expanded further. For detailed algorithms of STAMINA, readers are encouraged to read [22].
3 Results
This section presents results on the following case studies to illustrate the accuracy and efficiency of STAMINA: a genetic toggle switch [21, 23]; the following examples from the PRISM benchmark suite [16]: grid world robot, cyclic server polling system, and tandem queuing network; and the Jackson queuing network from INFAMY case studies [1]. All case studies are evaluated on STAMINA and INFAMY, except the genetic toggle switch ^{1}^{1}1INFAMY generates arithmetic errors on the genetic toggle switch model.. Experiments are performed on a 3.2 GHz AMD Debian Linux PC with six cores and 64 GB of RAM. For all experiments, the maximal number of iterations is set to , and the reduction factor is set to . All experiments terminate due to , where , before they reach . STAMINA is freely available at: https://github.com/formalverificationresearch/stamina.
We compare the runtime, state size, and verification results between STAMINA and INFAMY using the same precision . For all tables in this section, column reports the probability estimate threshold used to terminate state generation in STAMINA. The state space size is listed in column , where indicates one thousand states. Column reports the state space construction (C) and analysis (A) time in seconds. For STAMINA, the total construction and analysis time is the cumulation of runtime for all values for a model configuration. Columns and list the lower and upper probability bounds for the property under verification, and column lists the single probability value (within the precision ) reported by INFAMY. We select the best runtime reported by three configurations of INFAMY. The improvement in state size (column ) and runtime (column ) are represented by the ratio of state count generated by INFAMY to that of STAMINA (higher is better) and percentage improvement in runtime (higher is better), respectively.
Genetic toggle switch. The genetic toggle switch circuit model has two inputs, aTc and IPTG. It can be set to the OFF state by supplying it with aTc and can be set to the ON state by supplying it with IPTG [21]. Two important properties for a toggle switch circuit are the response time and the failure rate. The first experiments set IPTG to to measure the toggle switch’s response time. It should be noted that the input value of 100 molecules of IPTG is chosen to ensure that the circuit switches to the ON state. The later experiments initialize IPTG to to compute the failure rate, i.e., the probability that the circuit changes state erroneously within a cell cycle of seconds (an approximation of the cell cycle in E. coli [25]). Initially, LacI is set to and TetR is set to for both experiments. The CSL property used for both experiments, , describes the probability of the circuit switching to the ON state within a cell cycle of seconds. The ON state is defined as LacI below 20 and TetR above 40 molecules.
STAMINA  
Remark  
Property  
Guided  

Property  
Agnostic  

Property  
Guided  

Property  
Agnostic  

The propertyagnostic state space is generated with the probability estimate threshold . Table 1 shows large probability bounds: for and for . It is obvious that they are significantly inaccurate w.r.t. the precision of . The is then reduced to and state generation switches to the propertyguided state expansion mode, where the CSL property is used to guide state exploration, based on the previous state graph. Each state expansion step reduces the value by a factor of . To measure the effectiveness of the propertyguided state expansion approach, we compare state graphs generated with and without the propertyguided state expansion, as indicated by the “property agnostic” and “property guided” rows in the table. Propertyguided state expansion reduces the size of the state space without losing the analysis precision for the same value of . Specifically, the state expansion approach reduces the state space by almost for the response rate experiment.
Robot World. This case study considers a robot moving in an by grid and a janitor moving in a larger grid by, where the constant is used to significantly scale up the state space. The robot starts from the bottom left corner to reach the top right corner. The janitor moves around randomly. Either the robot or janitor can occupy one grid location at any given time. The robot also randomly communicates with the base station. The property of interest is the probability that the robot reaches the top right corner within time units while periodically communicating with the base station, encoded as .
Table 2 provides a comparison of results for and . For smaller grid size i.e, 32by32, the robot can reach the goal with a high probability of . Where as for a larger value of and , the robot is not able to reach the goal with considerable probability. STAMINA generates precise results that are similar to INFAMY, while exploring less than half of states with shorter runtime.
Model  Params  STAMINA  INFAMY  Improvement  












Robot 






   

  


   

  
Jackson 








Polling 









Tandem 







Jackson Queuing Network. A Jackson queuing network consists of interconnected nodes (queues) with infinite queue capacity. Initially, all queues are considered empty. Each station is connected to a single server which distributes the arrived jobs to different stations. Customers arrive as a Poisson stream with intensity for queues. The model is taken from [14, 11]. We compute the probability that, within 10 time units, the first queue has more that jobs and the second queue has more than jobs, given by .
Table 2 summarizes the results for this model. STAMINA uses roughly equal time to construct and analyze the model for , whereas INFAMY takes significantly longer to construct the state space, making it slower in overall runtime. For , STAMINA is faster in generating verification results In both configurations, STAMINA only explores approximately one third of the states explored by INFAMY.
Cyclic Server Polling System. This case study is based on a cyclic server attending stations. We consider the probability that station one is polled within 10 time units, . Table 2 summarizes the verification results for . The probability of station one being polled within seconds is for all configurations. Similar to previous case studies, STAMINA explores significantly smaller state space. The advantage of STAMINA in terms of runtime starts to manifest as the size of model (and hence the state space size) grows.
Tandem Queuing Network. A tandem queuing network is the simplest interconnected queuing network of two finite capacity () queues with one server each [19]. Customers join the first queue and enter the second queue immediately after completing the service. This paper considers the probability that the first queue becomes full in time units, depicted by the CSL property .
As seen in Table 2, there is almost fifty percent probability that the first queue is full in seconds irrespective of the queue capacity. As in the polling server, STAMINA explores significantly smaller state space. The runtime is similar for model with smaller queue capacity (). But the runtime improves as the queue capacity is increased.
4 Conclusions
This paper presents an infinitestate stochastic model checker, STAMINA, that uses path probability estimates to generate states with high probability and truncate unlikely states based on a specified threshold. Initial state construction is property agnostic, and the state space is used for stochastic model checking of a given CSL property. The calculated probability forms a lower and upper bound on the probability for the CSL property, which is guaranteed to include the actual probability. Next, if finer precision of the probability bound is required, it uses a propertyguided state expansion technique to explore states to tighten the reported probability range incrementally. Implementation of STAMINA is built on top of the PRISM model checker with tight integration to its API. Performance and accuracy evaluation is performed on case studies taken from various application domains, and shows significant improvement over the stateofart infinitestate stochastic model checker INFAMY. For future work, we plan to investigate methods to determine the reduction factor onthefly based on the probability bound. Another direction is to investigate heuristics to further improve the propertyguided state expansion, as well as, techniques to dynamically remove unlikely states.
Acknowledgment
Chris Myers is supported by the National Science Foundation under CCF1748200. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF.
References
 [1] https://depend.cs.unisaarland.de/tools/infamy/casestudies/

[2]
Andreychenko, A., Mikeev, L., Spieler, D., Wolf, V.: Parameter identification for markov models of biochemical reactions. In: Gopalakrishnan, G., Qadeer, S. (eds.) Computer Aided Verification. pp. 83–98. Springer Berlin Heidelberg, Berlin, Heidelberg (2011)
 [3] Aspnes, J., Herlihy, M.: Fast randomized consensus using shared memory. J. Algorithms 11(3), 441–461 (Sep 1990)
 [4] Aziz, A., Sanwal, K., Singhal, V., Brayton, R.: Modelchecking continuoustime markov chains. ACM Trans. Comput. Logic 1(1), 162–170 (Jul 2000)
 [5] Donaldson, A.F., Miller, A.: Symmetry reduction for probabilistic model checking using generic representatives. In: Proceedings of the 4th International Conference on Automated Technology for Verification and Analysis. pp. 9–23. ATVA’06, SpringerVerlag, Berlin, Heidelberg (2006)
 [6] Fecher, H., Leucker, M., Wolf, V.: Don’t know in probabilistic systems. In: Proceedings of the 13th International Conference on Model Checking Software. pp. 71–88. SPIN’06, SpringerVerlag, Berlin, Heidelberg (2006)
 [7] Fisler, K., Vardi, M.Y.: Bisimulation and model checking. In: Proceedings of the 10th IFIP WG 10.5 Advanced Research Working Conference on Correct Hardware Design and Verification Methods. pp. 338–341. CHARME ’99, SpringerVerlag, London, UK, UK (1999)
 [8] Fisler, K., Vardi, M.Y.: Bisimulation minimization and symbolic model checking. Form. Methods Syst. Des. 21(1), 39–78 (Jul 2002)
 [9] Groesser, M., Baier, C.: Partial order reduction for markov decision processes: A survey. In: Proceedings of the 4th International Conference on Formal Methods for Components and Objects. pp. 408–427. FMCO’05, SpringerVerlag, Berlin, Heidelberg (2006)
 [10] Hahn, E.M., Hermanns, H., Wachter, B., Zhang, L.: Infamy: An infinitestate markov model checker. In: Proceedings of the 21st International Conference on Computer Aided Verification. pp. 641–647. CAV ’09, SpringerVerlag, Berlin, Heidelberg (2009)
 [11] Hahn, E.M., Hermanns, H., Wachter, B., Zhang, L.: INFAMY: An infinitestate markov model checker. In: CAV. pp. 641–647 (2009)
 [12] Hasenauer, J., Wolf, V., Kazeroonian, A., Theis, F.J.: Method of conditional moments (mcm) for the chemical master equation. Journal of Mathematical Biology 69(3), 687–735 (Sep 2014). https://doi.org/10.1007/s0028501307115, https://doi.org/10.1007/s0028501307115
 [13] Hermanns, H., Wachter, B., Zhang, L.: Probabilistic CEGAR. In: Proceedings of the 20th International Conference on Computer Aided Verification. pp. 162–175. CAV ’08, SpringerVerlag, Berlin, Heidelberg (2008)
 [14] Jackson, J.: Networks of Waiting Lines. Operations Research 5, 518–521 (1957)
 [15] Katoen, J.P., Klink, D., Leucker, M., Wolf, V.: Threevalued abstraction for continuoustime markov chains. In: Proceedings of the 19th International Conference on Computer Aided Verification. pp. 311–324. CAV’07, SpringerVerlag, Berlin, Heidelberg (2007)
 [16] Kwiatkowsa, M., Norman, G., Parker, D.: The prism benchmark suite. In: Quantitative Evaluation of Systems, International Conference on(QEST). vol. 00, pp. 203–204 (09 2012). https://doi.org/10.1109/QEST.2012.14, doi.ieeecomputersociety.org/10.1109/QEST.2012.14
 [17] Kwiatkowska, M., Norman, G., Parker, D.: Symmetry reduction for probabilistic model checking. In: Proceedings of the 18th International Conference on Computer Aided Verification. pp. 234–248. CAV’06, SpringerVerlag, Berlin, Heidelberg (2006)
 [18] Kwiatkowska, M., Norman, G., Parker, D.: Stochastic Model Checking, pp. 220–270. Springer Berlin Heidelberg, Berlin, Heidelberg (2007)
 [19] Kwiatkowska, M., Norman, G., Parker, D.: Prism 4.0: Verification of probabilistic realtime systems. In: Proceedings of the 23rd International Conference on Computer Aided Verification. pp. 585–591. CAV’11, SpringerVerlag, Berlin, Heidelberg (2011)
 [20] Lapin, M., Mikeev, L., Wolf, V.: Shave: Stochastic hybrid analysis of markov population models. In: Proceedings of the 14th International Conference on Hybrid Systems: Computation and Control. pp. 311–312. HSCC ’11, ACM, New York, NY, USA (2011)
 [21] Madsen, C., Zhang, Z., Roehner, N., Winstead, C., Myers, C.: Stochastic model checking of genetic circuits. J. Emerg. Technol. Comput. Syst. 11(3), 23:1–23:21 (Dec 2014). https://doi.org/10.1145/2644817, http://doi.acm.org/10.1145/2644817
 [22] Neupane, T.: STAMINA: STochastic Approximate Modelchecker for INfinitestate Analysis. Master’s thesis, Utah State University (May 2019)

[23]
Neupane, T., Zhang, Z., Madsen, C., Zheng, H., Myers, C.J.: Approximation Techniques for Stochastic Analysis of Biological Systems. In: Pietro Lió, P.Z. (ed.) Automated Reasoning for Systems Biology and Medicine, Computational Biology, vol. 30, chap. 12, p. 480. Springer International Publishing, 1 edn. (Sep 2019).
https://doi.org/10.1007/9783030172978_12, https://doi.org/10.1007/9783030172978_12  [24] Parker, D.: Implementation of Symbolic Model Checking for Probabilistic Systems. Ph.D. thesis, University of Birmingham (2002)
 [25] Zheng, H., Ho, P.Y., Jiang, M., Tang, B., Liu, W., Li, D., Yu, X., Kleckner, N.E., Amir, A., Liu, C.: Interrogating the escherichia coli cell cycle by cell dimension perturbations. Proceedings of the National Academy of Sciences 113(52), 15000–15005 (2016)
Comments
There are no comments yet.