Model checking algorithms  are the cornerstone of computer-aided verification. As their input consists of both the system under verification and a logical formula that describes the property to be verified, they uniformly solve a wide range of verification problems, such as all verification problems expressible in linear-time temporal logic (LTL), computation-tree logic (CTL), or the modal -calculus. Recently, there has been a lot of interest in extending model checking from standard trace and tree properties to information flow policies like observational determinism or quantitative information flow. Such policies are called hyperproperties  and can be expressed in HyperLTL , an extension of LTL with trace quantifiers and trace variables. For example, observational determinism , the requirement that any pair of traces that have the same observable input also have the same observable output, can be expressed as the following HyperLTL formula: . For many information flow policies of interest, including observational determinism, there is no longer a need for property-specific algorithms: it has been shown that the standard HyperLTL model checking algorithm  performs just as well as a specialized algorithm for the respective property.
The class of hyperproperties studied in this paper is one where, by contrast, the standard model checking algorithm performes badly. We are interested in quantitative hyperproperties, i.e., hyperproperties that express a bound on the number of traces that may appear in a certain relation. A prominent example of this class of properties is quantitative non-interference [43, 45], where we allow some flow of information but, at the same time, limit the amount of information that may be leaked. Such properties are used, for example, to describe the correct behavior of a password check, where some information flow is unavoidable (“the password was incorrect”), and perhaps some extra information flow is accceptable (“the password must contain a special character”), but the information should not suffice to guess the actual password. In HyperLTL, quantitative non-interference can be expressed  as the formula The formula states that there do not exist traces (corresponding to more than bits of information) with the same observable input but different observable output. The bad performance of the standard model checking algorithm is a consequence of the fact that the traces are tracked simultaneously. For this purpose, the model checking algorithm builds and analyzes a -fold self-composition of the system.
We present a new model checking algorithm for quantitative hyperproperties that avoids the construction of the huge self-composition. The key idea of our approach is to use counting rather than checking as the basic operation. Instead of building the self-composition and then checking the satisfaction of the formula, we add new atomic propositions and then count the number of sequences of evaluations of the new atomic propositions that satisfy the specification. Quantitative hyperproperties are expressions of the following form:
where . The universal quantifiers introduce a set of reference traces against which other traces can be compared. The formulas and are HyperLTL formulas. The counting quantifer counts the number of paths with different valuations of the atomic propositions that satisfy . The requirement that no more than bits of information are leaked is the following quantitative hyperproperty:
As we show in the paper, such expressions do not change the expressiveness of the logic; however, they allow us to express quantitative hyperproperties in exponentially more concise form. The counting-based model checking algorithm then maintains this advantage with a logarithmic counter, resulting in exponentially better performance in both time and space.
The viability of our counting-based model checking algorithm is demonstrated on a Max#SAT-based prototype implementation. For quantitative hyperproperties of intrest, such as bounded leakage of a password checker, our algorithm shows promising results, as it significantly outperforms existing model checking approaches.
1.1 Related Work
Quantitative information-flow has been studied extensively in the literature. See, for example, the following selection of contributions on this topic: [43, 34, 19, 14, 1, 32]. Multiple verification methods for quantitative information-flow were proposed for sequential systems. For example, with static analysis techniques , approximation methods , equivalence relations [22, 3], and randomized methods . Quantitative information-flow for multi-threaded programs was considered in .
The study of quantitative information-flow in a reactive setting gained a lot of attention recently after the introduction of hyperproperties  and the idea of verifying the self-composition of a reactive system  in order to relate traces to each other. There are several possibilities to measure the amount of leakage, such as Shannon entropy [24, 15, 37], guessing entropy [34, 3], and min-entropy . A classification of quantitative information-flow policies as safety and liveness hyperproperties was given in . While several verification techniques for hyperproperties exists [5, 31, 38, 42], the literature was missing general approaches to quantitative information-flow control. SecLTL  was introduced as first general approach to model check (quantitative) hyperproperties, before HyperLTL , and its corresponding model checker , was introduced as a temporal logic for hyperproperties, which subsumes the previous approaches.
Using counting to compute the number of solutions of a given formula is studied in the literature as well and includes many probabilistic inference problems, such as Bayesian net reasoning , and planning problems, such as computing robustness of plans in incomplete domains . State-of-the-art tools for propositional model counting are Relsat  and c2d . Algorithms for counting models of temporal logics and automata over infinite words have been introduced in [27, 28, 44]. The counting of projected models, i.e., when some parts of the models are irrelevant, was studied in , for which tools such as #CLASP  and DSharp_P [2, 41] exist. Our SAT-based prototype implementation is based on a reduction to a Max#SAT  instance, for which a corresponding tool exists.
, which estimates the amount of leakage in Java programs,Moped-QLeak  is closest to our approach. However, their approach of computing a symbolic summary as an Algebraic Decision Diagram is, in contrast to our approach, solely based on model counting, not maximum model counting.
HyperLTL  extends linear-time temporal logic (LTL) with trace variables and trace quantifiers. Let be a set of atomic propositions. A trace is an infinite sequence over subsets of the atomic propositions. We define the set of traces . A subset is called a trace property and a subset is called a hyperproperty. We use the following notation to manipulate traces: let be a trace and be a natural number. denotes the -th element of . Therefore, represents the starting element of the trace. Let and . denotes the sequence . denotes the infinite suffix of starting at position .
Let be an infinite supply of trace variables. The syntax of HyperLTL is given by the following grammar:
where is an atomic proposition and is a trace variable. Note that atomic propositions are indexed by trace variables. The quantification over traces makes it possible to express properties like “on all traces must hold”, which is expressed by . Dually, one can express that “there exists a trace such that holds”, which is denoted by . The derived operators , , and are defined as for LTL. We abbreviate the formula , expressing that the traces and are equal with respect to a set of atomic propositions, by . Furthermore, we call a trace variable free in a HyperLTL formula if there is no quantification over and we call a HyperLTL formula closed if there exists no free trace variable in .
A HyperLTL formula defines a hyperproperty, i.e., a set of sets of traces. A set of traces satisfies the hyperproperty if it is an element of this set of sets. Formally, the semantics of HyperLTL formulas is given with respect to a trace assignment from to , i.e., a partial function mapping trace variables to actual traces. denotes that is mapped to , with everything else mapped according to . denotes the trace assignment that is equal to for all .
We say a set of traces satisfies a HyperLTL formula if , where is the empty trace assignment.
2.2 System model
A Kripke structure is a tuple consisting of a set of states , an initial state , a transition function , a set of atomic propositions , and a labeling function , which labels every state with a set of atomic propositions. We assume that each state has a successor, i.e., . This ensures that every run on a Kripke structure can always be extended to an infinite run. We define a path of a Kripke structure as an infinite sequence of states such that is the initial state of and for every . We denote the set of all paths of that start in a state with . Furthermore, denotes the set of all path prefixes and the set of all path suffixes. A trace of a Kripke structure is an infinite sequence of sets of atomic propositions , such that is the initial state of and for every . We denote the set of all traces of that start in a state with . We say that a Kripke structure satisfies a HyperLTL formula if its set of traces satisfies , i.e., if , where is the empty trace assignment.
2.3 Automata over infinite words
In our construction we use automata over infinite words. A Büchi automaton is a tuple , where is a set of states, is a set of initial states, is a transition relation, and are the accepting states. A run of on an infinite word is an infinite sequence of states, where and for each , . We define . A run is called accepting if . A word is accepted by and called a model of if there is an accepting run of on .
Furthermore, an alternating automaton, whose runs generalize from sequences to trees, is a tuple . , and are defined as above and being a transition function, which maps a state and a symbol into a Boolean combination of states. Thus, a run(-tree) of an alternating Büchi automaton on an infinite word is a -labeled tree. A word is accepted by and called a model if there exists a run-tree such that all paths trough are accepting, i.e., .
A strongly connected component (SCC) in is a maximal strongly connected component of the graph induced by the automaton. An SCC is called accepting if one of its states is an accepting state in .
3 Quantitative Hyperproperties
Quantitative Hyperproperties are properties of sets of computation traces that express a bound on the number of traces that may appear in a certain relation. In the following, we study quantitative hyperproperties that are specified in terms of HyperLTL formulas. We consider expressions of the following general form:
Both the universally quantified variables and the variable after the counting operator are trace variables; is a HyperLTL formula over atomic propositions and free trace variables ; is a set of atomic propositions; is a HyperLTL formula over atomic propositions and free trace variables and, additionally . The operator is a comparison operator; and is a natural number.
For a given set of traces and a valuation of the trace variables , the term computes the number of traces in that differ in their valuation of the atomic propositions in and satisfy . The expression is iff the resulting number satisfies the comparison with . Finally, the complete expression is iff for all combinations of traces in that satisfy , the comparison is satisfied.
Example 1 (Quantitative non-interference)
Quantitative information-flow policies [30, 13, 34, 20] allow the flow of a bounded amount of information. One way to measure leakage is with min-entropy , which quantifies the amount of information an attacker can gain given the answer to a single guess about the secret. The bounding problem  for min-entropy is to determine whether that amount is bounded from above by a constant , corresponding to
bits. We assume that the program whose leakage is being quantified is deterministic, and assume that the secret input to that program is uniformly distributed. The bounding problem then reduces to determining that there is no tuple ofdistinguishable traces [43, 45]. Let be the set of observable outputs. A simple quantitative information flow policy is then the following quantitative hyperproperty, which bounds the number of distinguishable outputs to , corresponding to a bound of bits of information:
A slightly more complicated information flow policy is quantitative non-interference. In quantitative non-interference, the bound must be satisfied for every individual input. Let be the observable inputs to the system. Quantitative non-interference is the following quantitative hyperproperty111We write short for where is the -projection of :
For each trace in the system, the property checks whether there are more than traces that have the same observable input as but different observable output.
Example 2 (Deniability)
A program satisfies deniability (see, for example, [7, 10]) when there is no proof that a certain input occurred from simply observing the output, i.e., given an output of a program one cannot derive the input that lead to this output. A deterministic program satisfies deniability when each output can be mapped to at least two inputs. A quantitative variant of deniability is when we require that the number of corresponding inputs is larger than a given threshold. Quantitative deniability can be specified as the following quantitative Hyperproperty:
For all traces of the system we count the number of sequences in the system with different input sequences and the same output sequence of , i.e., for the fixed output sequence given by we count the number of input sequences that lead to this output.
4 Model Checking Quantitative Hyperproperties
We present a model checking algorithm for quantitative hyperproperties based on model counting. The advantage of the algorithm is that its runtime complexity is independent of the bound and thus avoids the -fold self-composition necessary for any encoding of the quantitative hyperproperty in HyperLTL.
Before introducing our novel counting-based algorithm, we start by a translation of quantitative hyperproperties into formulas in HyperLTL and establishing an exponential lower bound for its representation.
4.1 Standard model checking algorithm: encoding quantitative hyperproperties in HyperLTL
The idea of the reduction is to check a lower bound of traces by existentially quantifying over traces, and to check an upper bound of traces by universally quantifying over traces. The resulting HyperLTL formula can be verified using the standard model checking algorithm for HyperLTL .
Every quantitative hyperproperty can be expressed as a HyperLTL formula. For , the HyperLTL formula has universal trace quantifiers in addition to the quantifiers in and . For , the HyperLTL formula has universal trace quantifiers and existential trace quantifiers in addition to the quantifiers in and . For , the HyperLTL formula has universal trace quantifiers and existential trace quantifiers in addition to the quantifiers in and .
For , we encode the quantitative hyperproperty as the following HyperLTL formula:
where is the HyperLTL formula with all occurrences of replaced by . The formula states that there is no tuple of traces different in the evaluation of , that satisfy . In other words, for every tuple of traces that differ in the evaluation of , one of the paths must violate . For , we use the same formula, with instead of .
For , we encode the quantitative hyperproperty analogously as the HyperLTL formula
The formula states that there exist paths that differ in the evaluation of and that all satisfy . For , we use the same formula, with instead of . Lastly, for , we encode the quantitative hyperproperty as a conjunction of the encodings for and for .
Example 3 (Quantitative non-interference in HyperLTL)
As discussed in Example 1, quantitative non-interference is the quantitative hyperproperty
where we measure the amount of leakage with min-entropy . The bounding problem for min-entropy asks whether the amount of information leaked by a system is bounded by a constant where is the number of bits. This is encoded in HyperLTL as the requirement that there are no traces distinguishable in their output:
This formula is equivalent to the formalization of quantitative non-interference given in .
Model checking quantitative hyperproperties via the reduction to HyperLTL is very expensive. In the best case, when , does not contain existential quantifiers, and does not contain universal quantifiers, we obtain an HyperLTL formula without quantifier alternations, where the number of quantifiers grows linearly with the bound . For quantifiers, the HyperLTL model checking algorithm  constructs and analyzes the -fold self-composition of the Kripke structure. The running time of the model checking algorithm is thus exponential in the bound. If , the encoding additionally introduces a quantifier alternation. The model checking algorithm checks quantifier alternations via a complementation of Büchi automata, which adds another exponent, resulting in an overall doubly exponential running time.
The model checking algorithm we introduce in the next section avoids the -fold self-composition needed in the model checking algorithm of HyperLTL and its complexity is independent of the bound .
4.2 Counting-based model checking algorithm
A Kripke structure violates a quantitative hyperproperty
if there is a -tuple of traces that satisfies the formula
where is the negation of the comparison operator . The tuple then satisfies the property and the number of -tuples for that satisfy and differ pairwise in the -projection of satisfies the comparison (The -projection of a sequence is defined as the sequence , such that for every position and every it holds that if and only if ). The tuples can be captured by the automaton composed of the product of an automaton that accepts all of traces that satisfy both and and a -self composition of . Each accepting run of the product automaton presents traces of that satisfy . On top of the product automaton, we apply a special counting algorithm which we explain in detail in Section 4.4 and check if the result satisfies the comparison .
Algorithm 1 gives a general picture of our model checking algorithm. The algorithm has two parts. The first part applies if the relation is one of . In this case, the algorithm checks whether a sequence over (propositions in ) corresponds to infinitely many sequences over . This is done by checking whether the product automaton has a so-called doubly pumped lasso(DPL), a subgraph with two connected lassos, with a unique sequence over and different sequences over . Such a doubly pumped lasso matches the same sequence over with infinitely many sequences over (more in section 4.4). If no doubly pumped lasso is found, a projected model counting algorithm is applied in the second part of the algorithm in order to compute either the maximum or the minimum value, corresponding to the comparison operator . In the next subsections, we explain the individual parts of the algorithm in detail.
4.3 Büchi automata for quantitative hyperproperties
For a quantitative hyperproperty and a Kripke structure , we first construct an alternating automaton for the HyperLTL property . Let and be alternating automata for subformulas and . Let where are all indexed atomic propositions that appear in . is constructed using following rules222The construction follows the one presented in  with a slight modification on the labeling of transitions. Labeling over atomic proposition instead of the states of the Kripke structure suffices, as any nondeterminism in the Kripke structure is inherently resolved, because we quantify over trace not paths:
|and when for|
|and when for|
|and when for|
|and when for|
For a quantified formula , we construct the product automaton of the Kripke structure and the Büchi automaton of . Here we reduce the alphabet of the automaton by projecting all atomic proposition in away:
Given the Büchi automaton for the hyperproperty it remains to construct the product with the -self composition of . The transitions of the automaton are defined over labels from where . . This is necessary to identify which transition was taken in each copy of , thus, mirroring a tuple of traces in . For each of the variables and we use following rule:
Finally, we transform the resulting alternating automaton to an equivalent Büchi automaton following the construction of Miyano and Hayashi .
4.4 Counting models of -Automata
Computing the number of words accepted by a Büchi automaton can be done by examining its accepting lassos. Consider, for example, the Büchi automata over the alphabet in Figure 1. The automaton on the left has one accepting lasso and thus has only one model, namely . The automaton on the right has infinitely many accepting lassos that accept infinitely many different words all of the from .
Computing the models of a Büchi automaton is insufficient for model checking quantitative hyperproperties as we are not interested in the total number of models. We rather maximize, respectively minimize, over sequences of subsets of atomic propositions the number of projected models of the Büchi automaton. For instance, consider the automaton given in Figure 2.
The automaton has infinitely many models. However, the maximum number of sequences that correspond to accepting lassos in the automaton with a unique sequence is two: For example, let be a natural number. For any model of the automaton and for each sequence the automaton accepts the following two sequences: and . Formally, given a Büchi automaton over and a set , such that , an -projected model (or projected model over ) is defined as a sequence that results in the -projection of an accepting sequence .
In the following, we define the maximum model counting problem over automata and give an algorithm for solving the problem. We show how to use the algorithm for model checking quantitative hyperproperties.
Definition 1 (Maximum Model Counting over Automata (MMCA))
Given a Büchi automaton over an alphabet for some set of atomic propositions AP and sets the maximum model counting problem is to compute
where is the point-wise union of and .
As a first step in our algorithm, we show how to check whether the maximum model count is equal to infinity.
Definition 2 (Doubly Pumped Lasso)
For a graph , a doubly pumped lasso in is a subgraph that entails a cycles and another different cycle that is reachable from .
In general, we distinguish between two types of doubly pumped lassos as shown in Figure 3. We call the lassos with periods and the lassos of the doubly pumped lasso. A doubly pumped lasso of a Büchi automaton is one in the graph structure of . The doubly pumped lasso is called accepting when has an accepting state. A more generalized formalization of this idea is given in the following theorem.
Let be a Büchi automaton for some set of atomic propositions and let . The automaton has infinitely many -projected models with if and only if has an accepting doubly pumped lasso with lassos and such that: 1) is an accepting lasso 2) 3) The period of shares at least one state with and 4) .
To check whether there is a sequence such that the number of -projected models of with is infinite, we search for a doubly pumped lasso satisfying the constraints given in Theorem 4.2. This can be done by applying the following procedure:
Given a Büchi automaton and sets , we construct the following product automaton where: , , and . The automaton has infinitely many models if there is an accepting lasso in such that: , i.e., has lassos and that share a state in the period of and , i.e., the lassos differ in the evaluation of in a position after the shared state and thus allows infinitely many different sequence over for the a sequence over . The lasso simulates a doubly pumped lasso in satisfying the constraints of Theorem 4.2.
Given an alternating Büchi automaton for a set of atomic propositions , the problem of checking whether there is a sequence such that has infinitely many -projected models with is Pspace-complete.
The lower and upper bound for the problem can be given by a reduction from and to the satisfiability problem of LTL . Due to the finite structure of Büchi automata, if the number of models of the automaton exceed the exponential bound , where is the set of states, then the automaton has infinitely many models.
For any Büchi automaton , the number of models of is less or equal to otherwise it is .
Assume the number of models is larger than then there are more than accepting lassos in . By the pigeonhole principle, two of them share the same -prefix. Thus, either they are equal or we found doubly pumped lasso in .
Let a Büchi automaton over a set of atomic propositions AP and sets . For each sequence the number of -projected models with is less or equal than otherwise it is .
From Corollary 1, we know that if no sequence matches to infinitely many -projected models then the number of such models is bound by . Each of these models has a run in which ends in an accepting strongly connected component. Also from Corollary 1, we know that every model has a lasso run of length . For each finite sequence of length that reaches an accepting strongly connected component, we count the number -projected words of length with and that end in an accepting strongly connected component. This number is equal to the maximum model counting number.
Algorithm 2 describes the procedure. An algorithm for the minimum model counting problem is defined in similar way. The algorithm works in a backwards fashion starting with states of accepting strongly connected components. In each iteration , the algorithm maps each state of the automaton with -projected words of length that reach an accepting strongly connected component. After iterations, the algorithm determines from the mapping of initial state a -projected word of length with the maximum number of matching -projected words.
The decisional version of the maximum model counting problem over automata (MMCA), i.e. the question whether the maximum is greater than a given natural number , is in .
Let a Büchi automaton over an alphabet for a set of atomic propositions and sets and a natural number
be given. We construct a nondeterministic Turing Machinewith access to a -oracle as follows: guesses a sequence . It then queries the oracle, to compute a number , such that , which is a problem . It remains to check whether . If so, accepts.
The following theorem summarizes the main findings of this section, which establish, depending on the property, an exponentially or even doubly exponentially better algorithm (in the quantitative bound) over the existing model checking algorithm for HyperLTL.
Given a Kripke structure and a quantitative hyperproperty with bound , the problem whether can be decided in logarithmic space in the quantitative bound and in polynomial space in the size of .
5 A Max#Sat-based Approach
For existential HyperLTL formulas and , we give a more practical model checking approach by encoding the automaton-based construction presented in Section 4 into a propositional formula.
Given a Kripke structure and a quantitative hyperproperty over a set of atomic propositions and bound , our algorithm constructs a propositional formula such that, every satisfying assignment of uniquely encodes a tuple of lassos