1 Introduction
While model checking algorithms and tools (e.g., [18, 17, 25, 12, 54, 46]) have, in the past, focused on trace properties, recent failures in securitycritical systems, such as Heartbleed [27], Meltdown [58], Spectre [51], or Log4j [1], have triggered the development of model checking algorithms for properties that relate multiple computation traces to each other, i.e., hyperproperties [21]. Although the counterexample returned by such a model checker for hyperproperties, which takes the shape of a set of traces, may aid in the debugging process, understanding and narrowing down which features are actually responsible for the erroneous relation between the traces in the counterexample requires significantly more manual effort than for trace properties. In this paper, we develop an explanation technique for these more complex counterexamples that identifies the actual causes [43, 44, 45] of hyperproperty violations.
Existing hyperproperty model checking approaches (e.g., [32, 34, 48]), take a HyperLTL formula as an input. HyperLTL is a temporal logic extending LTL with explicit trace quantification [20].
For example, observational determinism, which requires that all traces agree on their observable outputs whenever they agree on their observable inputs , can be formalized in HyperLTL as In case a system model violates observational determinism, the model checker consequently returns a set of two execution traces witnessing the violation.
A first attempt in explaining model checking results of HyperLTL specifications has been made with HyperVis [47], which visualizes a counterexample returned by the model checker MCHyper [34] in a browser application. While the visualizations are already useful to analyze the counterexample at hand, it fails to identify causes for the violation in several securitycritical scenarios. This is because HyperVis identifies important atomic propositions that appear in the HyperLTL formula and highlights these in the trace and the formula. For detecting causes, however, this is insufficient: a cause for a violation of observational determinism, for example, could be a branch on the valuation of a secret input , which is not even part of the formula (see Sec. 3 for a running example).
Defining what constitutes an actual cause for an effect (a violation) in a given scenario is a precious contribution by Halpern and Pearl [43, 44, 45], who refined and formalized earlier approaches based on counterfactual reasoning [57]: Causes are sets of events such that, in the counterfactual world where they do not appear, the effect does not occur either. One of the main insights of Halpern and Pearl’s work, however, is that naive counterfactuals are too imprecise. If, for instance, our actual cause preempted another potential cause, the mere absence of the actual cause will not be enough to prevent the effect, which will be still produced by the other cause in the new scenario. Halpern and Pearl’s definition therefore allows to carefully control for other possible causes through the notion of contingencies. In the modified definition [43], contingencies allow to fix certain features of the counterfactual world to be exactly as they are in the actual world, regardless of the system at hand. Such a contingency effectively modifies the dynamics of the underlying model, and one insight of our work is that defining actual causality for reactive systems also needs to modify the system under a contingency. Notably, most works regarding trace causality [13, 38] do not consider contingencies but only counterfactuals, and thus are not able to find true actual causes.
In this paper, we develop the notion of actual causality for effects described by HyperLTL formulas and use the generated causes as explanations for counterexamples returned by a model checker. We show that an implementation of our algorithm is practically feasible and significantly increases the stateoftheart in explaining and analyzing HyperLTL model checking results.
2 Preliminaries
We model a system as a Moore machine [61] where is a finite set of states, is the initial state, is the set of atomic propositions consisting of inputs and outputs , is the transition function determining the successor state for a given state and set of inputs, and is the labeling function mapping each state to a set of outputs. A trace of is an infinite sequence of sets of atomic propositions with , where and for all . We usually write to refer to the set at the th position of . With , we denote the set of all traces of . For some sequence of inputs , the trace is defined by and for all . A trace property is a set of traces. A hyperproperty is a lifting of a trace property, i.e., a set of sets of traces. A model satisfies a hyperproperty if the set of traces of is an element of the hyperproperty, i.e., .
2.1 HyperLTL
HyperLTL is a recently introduced logic for expressing temporal hyperproperties, extending lineartime temporal logic (LTL) [63] with trace quantification:
We also consider the usual derived Boolean (, , ) and temporal operators (, , ). The semantics of HyperLTL formulas is defined with respect to a set of traces and a trace assignment that maps trace variables to traces. To update the trace assignment so that it maps trace variable to trace , we write .
We explain counterexamples found by MCHyper [34, 23], which is a model checker for HyperLTL formulas, building on ABC [12]. MCHyper takes as inputs a hardware circuit, specified in the Aiger format [8], and a HyperLTL formula. MCHyper solves the model checking problem by computing the selfcomposition [6] of the system. If the system violates the HyperLTL formula, MCHyper returns a counterexample. This counterexample is a set of traces through the original system that together violate the HyperLTL formula. Depending on the type of violation, this counterexample can then be used to debug the circuit or refine the specification iteratively.
2.2 Actual Causality
A formal definition of what actually causes an observed effect in a given context has been proposed by Halpern and Pearl [44]. Here, we outline the version later modified by Halpern [43]. Causality is defined with respect to a causal model , given by a signature and set of structural equations , which define the dynamics of the system. A signature is a tuple , where and are disjoint sets of variables, termed exogenous and endogenous variables, respectively; and defines the range of possible values for all variables . A context is an assignment to the variables in such that the values of the exogenous variables are determined by factors outside of the model, while the value of some endogenous variable is defined by the associated structural equation . An effect in a causal model is a Boolean formula over assignments to endogenous variables. We say that a context of a model satisfies a partial variable assignment for if the assignments in and in coincide for every variable . The extension for Boolean formulas over variable assignments is as expected. For a context and a partial variable assignment , we denote by the context in which the values of the variables in are set according to , and all other values are computed according to the structural equations.
The actual causality framework of Halpern and Pearl aims at defining what events (given as variable assignments) are the cause for the occurrence of an effect in a specific given context. We now provide the formal definition.
Definition 1 ([44, 43])
A partial variable assignment is an actual cause of the effect in if the following three conditions hold.

AC1: and , i.e., both cause and effect are true in the actual world.

AC2: There is a set of endogenous variables and an assignment to the variables in s.t. if , then .

AC3: is minimal, i.e. no subset of satisfies AC1 and AC2.
Intuitively, AC2 states that in the counterfactual world obtained by intervening on the cause in the actual world (that is, setting the variables in to ), the effect does not appear either. However, intervening on the possible cause might not be enough, for example when that cause preempted another. After intervention, this other cause may produce the effect again, therefore clouding the effect of the intervention. To address this problem, AC2 allows to reset values through the notion of contingencies, i.e., the set of variables can be reset to , which is (implicitly) universally quantified. However, since the actual world has to model , it is in fact uniquely determined. AC3, lastly, enforces the cause to be minimal by requiring that all variables in are strictly necessary to achieve AC1 and AC2. For an illustration of Halpern and Pearl’s actual causality, see Ex. 1 in Sec. 3.
3 Running Example
Consider a securitycritical setting with two security levels: a highsecurity level and a lowsecurity level . Inputs and outputs labeled as highsecurity, denoted by and respectively, are confidential and thus only visible to the user itself, or, e.g., admins. Inputs and outputs labeled as lowsecurity, denoted by and respectively, are public and are considered to be observable by an attacker.
Our system of interest is modeled by the state graph representation shown in Fig. 1, which is treated as a black box by an attacker. The system is run without any lowsecurity inputs, but branches depending on the given highsecurity inputs. If in one of the first two steps of an execution, a highsecurity input is encountered, the system outputs only the highsecurity variable directly afterwards and in the subsequent steps both outputs, regardless of inputs. If no highsecurity input is given in the first step, the lowsecurity output is enabled and after the second step, again both outputs are enabled, regardless of what input is fed into the system.
A prominent example hyperproperty is observational determinism from the introduction: which states that any sequence of lowinputs always produces the same lowoutputs, regardless of what the highsecurity level inputs are. . The formula states that all traces and must agree in the lowsecurity outputs if they agree in the lowsecurity inputs. Our system at hand does not satisfy observational determinism, because the lowsecurity outputs in the first two steps depend on the present highsecurity inputs. Running MCHyper, a model checker for HyperLTL, results in the following counterexample: and . With the same lowsecurity input (none) the traces produce different lowsecurity outputs by visiting or on the way to .
In this paper, our goal is to explain the violation of a HyperLTL formula on such a counterexample. Following Halpern and Pearl’s explanation framework [45], an actual cause that is considered to be possibly true or possibly false constitutes an explanation for the user. We only consider causes over input variables, which can be true and false in any model. Hence, finding an explanation amounts to answering which inputs caused the violation on a specific counterexample. Before we answer this question for HyperLTL and the corresponding counterexamples given by sets of traces (see Sec. 4), we first illustrate Halpern and Pearl’s actual causality (see Sec. 2.2) with the above running example.
Example 1
Finite executions of a system can be modeled in Halpern and Pearl’s causal models. Consider inputs as exogenous variables and outputs as endogenous variables . The indices model at which step of the execution the variable appears. We omit the inputs at the third position and the outputs at the first position because they are not relevant for the following exposition. We have that for every . Now, the following manually constructed structural equations encode the transitions: (1) , (2) , (3) and (4) . Consider context , effect , and candidate cause . Because of Eq. (1), we have that and , hence AC1 is satisfied. Regarding AC2, this example allows us to illustrate the need for contingencies to accurately determine the actual cause: If we only consider intervening on the candidate cause , we still have , because with and Eq. (3) it follows that . However, in the actual world, the second high input has no influence on the effect. We can control for this by considering the contingency , which is satisfied in the actual world, but not after the intervention on . Because of this contingency, we then have that , and hence, AC2 holds. Because a singleton set automatically satisfies AC3, we can infer that the first high input was the actual cause for any low output to be enabled in the actual world. Note that, intuitively, the contingency allows us to ignore some of the structural equations by ignoring the value they assign to in this context. Our definitions in Sec. 4 will allow similar modifications for counterexamples to hyperproperties.
4 Causality for Hyperproperty Violations
Our goal in this section is to formally define actual causality for the violation of a hyperproperty described by a general HyperLTL formula , observed in a counterexample to . Such a counterexample is given by a trace assignment to the trace variables appearing in . Note that, for universal quantifiers, the assignment of a single trace to the bounded variable suffices to define a counterexample. For existential quantifiers, this is not the case: to prove that an existential quantifier cannot be instantiated we need to show that no system trace satisfies the formula in its body, i.e., provide a proof for the whole system. In this work, we are interested in explaining violations of hyperproperties, and not proofs of their satisfaction [16]. Hence, we limit ourselves to instantiations of the outermost universal quantifiers of a HyperLTL formula, which can be returned by model checkers like MCHyper [34, 23]. Since our goal is to explain counterexamples, restricting ourselves to results returned by existing model checkers is reasonable. Note that MCHyper can still handle formulas of the form where is quantifier free, including interesting information flow policies like generalized noninterference [60]. The returned counterexample then only contains traces that instantiate the universal quantifiers, the existential quantifiers are not instantiated for the above reason. In the following, we restrict ourselves to formulas and counterexamples of this form.
Definition 2 (Counterexample)
Let be a transition system and denote , and let be a HyperLTL formula of the form , where is a HyperLTL formula that does not start with . A counterexample to in is a partial trace assignment such that .
For ease of notation, we sometimes refer to simply as the tuple of its instantiations . In terms of Halpern and Pearl’s actual causality as outlined in Sec. 2.2, a counterexample describes the actual world at hand, which we want to explain. As a next step, we need to define an appropriate language to reason about possible causes and contingencies in our counterexample. We will use sets of events, i.e., values of atomic propositions at a specific position of a specific trace in the counterexample.
Definition 3 (Event)
An event is a tuple such that or for some atomic proposition , is a point in time, and is a trace of a system . We say that a counterexample satisfies a set of events , and denote , if for every event the two following conditions hold:

for some , i.e., all events in reason about traces in ,

iff , i.e., holds on trace of the counterexample at time .
We assume that the set is a disjoint union of input an output propositions, that is, . We say that is an input event if , and we call it an output event if . We denote the set of input events by and the set of output events by . These events have a direct correspondence with the variables appearing in Halpern and Pearl’s causal models: we can identify input events with exogenous variables (because their value is determined by factors outside of the system) and output events with endogenous variables.
We define a cause as a set of input events, while an effect is a possibly infinite Boolean formula over . Note that, similar to [36], every HyperLTL formula can be represented as a first order formula over events, e.g. . For some set of events , let denote the set of atomic propositions defined positively by on trace at position . Dualy, we define .
In order to define actual causality for hyperproperties we need to formally define how we obtain the counterfactual executions under some contingency for the case of events on infinite traces. We define a contingency as a set of output events. Mapping Halpern and Pearl’s definition to transition systems, contingencies reset outputs in the counterfactual traces back to their value in the original counterexample, which amounts to changing the state of the system, and then following the transition function from the new state. For a given trace of the counterexample, we describe all possible behaviors under arbitrary contingencies with the help of a counterfactual automaton. The concrete contingency on a trace is defined by additional input variables. In the following, let be a set of auxiliary input variables expressing whether a contingency is invoked at the given step of the execution and be a function s.t. .
Definition 4 (Counterfactual Automaton)
Let be a system with , i.e., every state is uniquely labeled, and there exists a state for every combination of outputs. Let be a trace of in a finite, lassoshaped representation. The counterfactual automaton is defined as follows:

where if , else , and
, 
.
A counterfactual automaton is effectively a chain of copies of the original system, of the same length as the counterexample. An execution through the counterfactual automaton starts in the first copy corresponding to the first position in the counterexample trace, and then moves through the chain until it eventually loops back from copy to copy . A transition in the counterfactual automaton can additionally specify setting as a contingency some output variable if the auxiliary input variable is enabled. In this case, the execution will move to a state in the next automaton of the chain where all the outputs are as usual, except , which will have the same value as in the counterexample . Note that, under the assumption that all states of the original system are uniquely labeled and there exists a state for every combination of output variables, the function is uniquely determined. ^{1}^{1}1The same reasoning can be applied to arbitrary systems by considering for contingencies largest sets of outputs for which the assumption holds, with the caveat that the counterfactual automaton may model fewer contingencies. Consequently, computed causes may be less precise in case multiple causes appear in the counterexample. A counterfactual automaton for our running example is described in App. 0.A.1.
Next, we need to define how we intervene on a set of traces with a candidate cause given as a set of input events, and a contingency given as a set of output events. We define an intervention function, which transforms a trace of our original automaton to an input sequence of an counterfactual automaton.
Definition 5 (Intervention)
For a cause , a contingency and a trace , the function returns a trace such that for all the following holds: We lift the intervention function to counterexamples given as a tuple as follows:
Intuitively, the intervention function flips all the events that appear in the cause : If some appears positively in the candidate cause , it will appear negatively in the resulting input sequence, and viceversa. For a contingency , the intervention function enables their auxiliary input for the counterfactual automaton at the appropriate time point irrespective of their value, as the counterfactual automaton will take care of matching the atomic propositions value to the value in the original counterexample .
4.1 Actual Causality for HyperLTL Violations
We are now ready to formalize what constitutes an actual cause for the violation of a hyperproperty described by a HyperLTL formula.
Definition 6 (Actual Causality for HyperLTL)
Let be a counterexample to a HyperLTL formula in a system . The set is an actual cause for the violation of on if the following conditions hold.
 SAT

.
 CF

There exists a contingency and a nonempty subset such that: .
 MIN

is minimal, i.e., no subset of satisfies SAT and CF.
Unlike in Halpern and Pearl’s definition (see Sec. 2.2), the condition SAT requires to satisfy only the cause, as we already know that the effect , i.e., the violation of the specification, is satisfied by virtue of being a counterexample. CF is the counterfactual condition corresponding to AC2 in Halpern and Pearl’s definition, and it states that after intervening on the cause, under a certain contingency, the set of traces satisfies the property. (Note that we use a conjunction of two statements here while Halpern and Pearl use an implication. This is because they implicitly quantify universally over the values of the variables in the set (which should be as in the actual world) where in our setting the set of contingencies already defines explicit values.) MIN is the minimality criterion directly corresponding to AC3.
Example 2
Consider our running example from Sec. 3, i.e., the system from Fig. 1 and the counterexample to observational determinism . Let us consider what it means to intervene on the cause . Note that we have , hence the condition SAT is satisfied. For CF, let us first consider an intervention without contingencies. This results in . However, , because the low outputs of and differ at the third position: and . This is because now the second high input takes effect, which was preempted by the first cause in the actual counterexample. The contingency now allows us to control this by modyfing the state after taking the second high input as follows: . Note that is not a trace of the model depicted in Fig. 1, because there is no transition that explains the step from to . It is, however, a trace of the counterfactual automaton (see App. 0.A.1), which encodes the set of counterfactual worlds for the trace . The fact that we consider executions that are not part of the original system allows us to infer that only the first high input was an actual cause in our running example. Disregarding contingencies, we would need to consider both high inputs as an explanation for the violation of observational determinism, even though the second high input had no influence. Our treatment of contingencies corresponds directly to Halpern and Pearl’s causal models, which allow to ignore certain structural equations as outlined in Ex. 1.
Remark: With our definitions, we strictly generalize Halpern and Pearl’s actual causality to reactive systems modeled as Moore machines and effects expressed as HyperLTL formulas. Their structural equation models can be encoded in a onestep Moore machine; effect specifying a Boolean combination of primitive events can be encoded in the more expressive logic HyperLTL. Just like for Halpern and Pearl, our actual causes are not unique. While there can exist several different actual causes, the set of all actual causes is always unique. It is also possible that no actual cause exists: If the effect occurs on all system traces, there may be no actual cause on a given individual trace.
4.2 Finding Actual Causes with Model Checking
In this section, we consider the relationship between finding an actual cause for the violation of a HyperLTL formula starting with a universal quantifier and model checking of HyperLTL. We show that the problem of finding an actual cause can be reduced to a model checking problem where the generated formula for the model checking problem has one additional quantifier alternation. While there might be a reduction resulting in a more efficient encoding, our current result suggests that causality checking is the harder problem. The key idea of our reduction is to use counterfactual automata (that encode the given counterexample and the possible counterfactual traces) together with the HyperLTL formula described in the proof to ensure the conditions SAT, CF, and MIN on the witnesses for the model checking result.
Proposition 1
We can reduce the problem of finding an actual cause for the violation of an HyperLTL formula starting with a universal quantifier to the HyperLTL model checking problem with one additional quantifier alternation.
Proof
Let be a counterexample for the formula where is a HyperLTL formula that does not have a universal first quantifier. We provide the proof for the case of for readability reasons, but it can be extended to any natural number . We assume that have some regular representation, as otherwise the initial problem of computing causality is not well defined. That is, we denote such that .
In order to find an actual cause, we need to find a pair of traces that are counterfactuals for ; satisfy the property ; and the changes from to are minimal with respect to set containment. Changes in inputs between and in the loop part should reoccur in repeatedly. Note that the differences between the counterexample and the witness of the model checking problem encode the actual cause, i.e. in case of a difference, the cause contains the event that is present on the counterexample. To reason about these changes, we use the counterfactual automaton for each , which also allows us to search for the contingency as part of the input sequence of . Note that each consists of copies, that indicate in which step the automaton is with respect to and its loop . For , we label each state in with the additional label , to indicate that the system is now in the loop part of . In addition, we add to the initial state of the label , and we add to the initial state of the system the label . The formula below states that the trace begins its run from the initial state of (and thus stays in this component through the whole run), and that every time visits a state on the loop, the same input sequence is observed. This way we enforce the periodic input behavior of the traces on .
For a subset of locations and a subset of input propositions we define that states that differs from in at least all events for ; and the formula that states that for all events that are not defined by and , is equal to .
We now define the formula that states that the set of inputs (and locations) on which trace differs from is not contained in the corresponding set for . We only check locations up until the length of .
Denote where and are trace variables for . The formula described below states that the two traces and are part of the systems , and have the same loop structure as and , and satisfy . That is, these traces can be obtained by changing the original traces and avoid the violation.
Finally, described below states that the counterfactuals correspond to a minimal change in the input events with respect to . All other traces that the formula reasons about start at the initial state of the original system and thus are not affected by the counterfactual changes. We verify against the product automaton to find these traces that witness the presence of a cause, counterfactual and contingency.
Then, if there exists two such traces in the system , they correspond to a minimal cause for the violation. Otherwise, there are no traces of the counterfactual automata that can be obtained from using counterfactual reasoning and satisfy the formula . ∎
We have shown that we can use HyperLTL model checking to find an actual cause for the violation of a HyperLTL formula. The resulting model checking problem has an additional quantifier alternation which suggests that identifying actual causes is a harder problem. Therefore, we restrict ourselves to finding actual causes for violations of universal HyperLTL formulas. This keeps the algorithms we present in the next section practical as we start without any quantifier alternation and need to solve a model checking problem with a single quantifier alternation. While this restriction excludes some interesting formulas, many can be strengthened into this fragment such that we are able to handle close approximations (c.f. [24]). Any additional quantifier alternation from the original formula carries over to an additional quantifier alternation in the resulting model checking problem which in turn leads to an exponential blowup. The scalability of our approach is thus limited by the complexity of the model checking problem.
5 Computing Causes for Counterexamples
In this section, we describe our algorithm for finding actual causes of hyperproperty violations. Our algorithm is implemented on top of MCHyper [34], a model checker for hardware circuits and the alternationfree fragment of HyperLTL. In case of a violation, our analysis enriches the provided counterexample with the actual cause which can explain the reason for the violaiton to the user.
We first provide an overview of our algorithm and then discuss each step in detail. First, we compute an overapproximation of the cause using a satisfiability analysis over the transitions taken in the counterexample. This analysis results in a set of candidate events . As we show in Prop. 2, every actual cause for the violation is a subset of . In addition, in Prop. 3 we show that the set satisfies conditions SAT and CF. To ensure MIN, we search for the smallest subset that satisfies SAT and CF. This set is then our minimal and therefore actual cause.
To check condition CF, we need to check the counterfactual of each candidate cause , and potentially also look for contingencies for . We separate our discussion as follows. We first discuss the calculation of the overapproximation (Sec. 5.1), then we present the ActualCause algorithm that identifies a minimal subset of that is an actual cause (Sec. 5.2), and finally we discuss in detail the calculation of contingencies (Sec. 5.3).
In the following sections, we use a reduction of the universal fragment of HyperLTL to LTL, and the advantages of the linear translation of LTL to alternating automata, as we now briefly outline.
HyperLTL to LTL. Let be a HyperLTL formula and be the counterexample. We construct an LTL formula from as follows [30]: atomic propositions indexed with different trace variables are treated as different atomic propositions and trace quantifiers are eliminated. For example results in the LTL formula . As for , we use the same renaming in order to zip all traces into a single trace, for which we assume the finite representation , which is also the structure of the model checker’s output. The trace is a violation of the formula , i.e., satisfies . We denote . We can then assume, for implementation concerns, that the specification (and its violation) is an LTL formula, and the counterexample is a single trace. After our causal analysis, the translation back to a cause over hyperproperties is straightforward as we maintain all information about the different traces in the counterexample. Note that this translation works due to the synchronous semantics of HyperLTL.
Finite Trace Model Checking Using Alternating Automata. In verifying condition CF (that is, in computing counterfactuals and contingencies), we need to apply finite trace model checking, as we want to check if the modified trace in hand still violates the specification , that is, satisfies . To this end, we use the linear algorithm of [35], that exploits the linear translation of to an alternating automaton , and using backwards analysis checks the satisfaction of the formula. An alternating automaton [67] generalizes nondeterministic and universal automata, and its transition relation is a Boolean function over the states. The run of alternating automaton is then a tree run that captures the conjunctions in the formula. We use the algorithm of [35] as a black box (see App. LABEL:prelim:alt for a formal definition of alternating automata and App. LABEL:app:ltltoalt for the translation from LTL to alternating automata). For the computation of contingencies we use an additional feature of the algorithm of [35] – the algorithm returns an accepting run tree of on , with annotations of nodes that represent atomic subformulas of that take part in the satisfaction of . We use this feature also in Sec. 5.1 when calculating the set of candidate causes.
5.1 Computing the Set of Candidate Causes
The events that might have been a part of the cause to the violation are in fact all events that appear on the counterexample, or, equivalently, all events that appear in and . Note that due to the finite representation, this is a finite set of events. Yet, not all events in this set can cause the violation. In order to remove events that could not have been a part of the cause, we perform an analysis of the transitions of the system taken during the execution of . With this analysis we detect which events appearing in the trace locally cause the respective transitions, and thus might be part of the global cause. Events that did not trigger a transition in this specific trace cannot be a part of the cause. Note that causing a transition and being an actual cause are two different notions  actual causality is defined over the behaviour of the system, not on individual traces. We denote the overapproximation of the cause as . Formally, we represent each transition as a Boolean function over inputs and states. Let denote the formula representing the transition of the system taken when reading , and let be a Boolean variable that corresponds to the event .^{2}^{2}2That is, corresponds to the event . Recall that the atomic propositions on the zipped trace are annotated with the original trace from . Denote , that is, expresses the exact set of events in . In order to find events that might trigger the transition , we check for the unsatisfiable core of . Intuitively, the unsatisfiable core of is the set of events that force the system to take this specific transition. For every (or ) in the unsatisfiable core that is also a part of , we add (or ) to .
We use unsatisfiable cores in order to find input events that are necessary in order to take a transition. However, this might not be enough. There are cases in which inputs that appear in formula are not detected using this method, as they are not essential in order to take a transition; however, they might be considered a part of the actual cause, as negating them can avoid the violation. Therefore, as a second step, we apply the algorithm of [35] on the annotated automaton in order to find the specific events that affect the satisfaction of , and we add these events to . Then, the unsatisfiable core approach provides us with inputs that affect the computation and might cause the violation even though they do not appear on the formula itself; while the alternating automaton allows us to find inputs that are not essential for the computation, but might still be a part of the cause as they appear on the formula.
Proposition 2
The set is indeed an overapproximation of the cause for the violation. That is, every actual cause for the violation is a subset of .
Proof (sketch)
Let be an event such that is not in the unsatisfiable core of and does not directly affect the satisfaction of according to the alternating automata analysis. That is, the transition corresponding to is taken regardless of , and thus all future events on remain the same regardless of the valuation of . In addition, the valuation of the formula is the same regardless of , since: (1) does not directly affect the satisfaction of ; (2) does not affect future events on (and obviously it does not affect past events). Therefore, every set such that is not minimal, and does not form a cause. Since the above is true for all events , it holds that for every actual cause . ∎
Proposition 3
The set satisfies conditions SAT and CF.
Proof
The condition SAT is satisfied as we add to only events that indeed occur on the counterexample trace. For CF, consider that is a superset of the actual cause , so the same contingency and counterfactual of will also apply for . This is since in order to compute counterfactual we are allowed to flip any subset of the events in , and any such subset is also a subset of . In addition, in computing contingencies, we are allowed to flip any subset of outputs as long as they agree with the counterexample trace, which is independent in and . ∎
5.2 Checking Actual Causality
Due to Prop. 2 we know that in order to find an actual cause, we only need to consider subsets of as candidate causes. In addition, since satisfies condition SAT, so do all of its subsets. We thus only need to check conditions CF and MIN for subsets of . Our actual causality computation, presented in Alg. 1 is as follows. We start with the set , that satisfies SAT and CF. We then check if there exists a more minimal cause that satisfies CF. This is done by iterating over all subsets of , ordered by size and starting with the smallest ones, and checking if the counterfactual for the manages to avoid the violation; and if not, if there exists a contingency for this . If the answer to one of these questions is yes, then is a minimal cause that satisfies SAT, CF, and MIN, and thus we return as our actual cause. We now elaborate on CF and MIN.
CF. As we have mentioned above, checking condition CF is done in two stages – checking for counterfactuals and computing contingencies. We first show that we do not need to consider all possible counterfactuals, but only one counterfactual for each candidate cause.
Proposition 4
In order to check if a candidate cause is an actual cause it is enough to test the one counterfactual where all the events in are flipped.
Proof
Assume that there is a strict subset of such that we only need to flip the valuations of events in in order to find a counterfactual or contingency, thus satisfies CF. Since is a more minimal cause than , we will find it during the minimality check. ∎
We assume that CF holds for the input set and check if it holds for any smaller subset . CF holds for if (1) flipping all events in is enough to avoid the violation of or if (2) there exists a nonempty set of contingencies for that ensures that is not violated. The computation of contingencies is described in Alg. 2. Verifying condition CF involves model checking traces against an LTL formula, as we check in Alg. 1 (line 3) if the property is still violated on the counterfactual trace with the empty contingency, and on the counterfactual traces resulting from the different contingency sets we consider in Alg. 2 (line 7). In both scenarios, we apply finite trace model checking, as described at the beginning of Sec. 5 (as we assume a lassoshaped representation of our traces).
MIN. To check if is minimal, we need to check if there exists a subset of that satisfies CF. We check CF for all subsets, starting with the smallest one, and report the first subset that satisfies CF as our actual cause. (Note that we already established that and all of its subsets satisfy SAT.)
5.3 Computing Contingencies
Recall that the role of contingencies is to eliminate the effect of other possible causes from the counterfactual world, in case these causes did not affect the violation in the actual world. More formally, in computing contingencies we look for a set of output events such that changing these outputs from their value in the counterfactual to their value in the counterexample results in avoiding the violation. Note that the inputs remain as they are in the counterfactual. We note that the problem of finding contingencies is hard, and in general is equivalent to the problem of model checking. This is since we need to consider all traces that are the result of changing some subset of events (output + time step) from the counterfactual back to the counterexample, and to check if there exists a trace in this set that avoids the violation. Unfortunately, we are unable to avoid an exponential complexity in the size of the original system, in the worst case. However, our experiments show that in practice, most cases do not require the use of contingencies.
Our algorithm for computing contingencies (Alg. 2) works as follows. Let be the counterfactual trace. As a first step, we use the annotated run tree of the alternating automaton on to detect output events that appear in and take part in satisfying . Subsets of these output events are our first candidates for contingencies as they are directly related to the violation (Alg. 2 lines 22). If we were not able to find a contingency, we continue to check all possible subsets of output events that differ from the original counterexample trace. We test the different outputs by feeding the Counterfactual automaton of Def. 4 with additional inputs from the set . The resulted trace is then our candidate contingency, which we try to verify against . The number of different input sequences is bounded by the size of the product of the Counterfactual automaton and the automaton for , and thus the process terminates.
Theorem 5.1 (Correctness)
Our algorithm is sound and complete. That is, let be a counterexample with a finite representation to a HyperLTL formula . Then, our algorithm returns an actual cause for the violation, if such exists.
Proof
Soundness. Since we verify each candidate set of inputs according to the conditions SAT, CF and MIN, it holds that every output of our algorithm is indeed an actual cause. Completeness. If there exists a cause, then due to Prop. 2, it is a subset of the finite set . Since in the worst case we test every subset of , if there exists a cause we will eventually find it. ∎
6 Implementation and Experiments
We implemented Alg. 1 and evaluated it on publicly available example instances of HyperVis [47], for which their state graphs were available. In the following, we provide implementation details, report on the running time and show the usefulness of the implementation by comparing to the highlighting output of HyperVis. Our implementation is written in Python and uses pyaiger [68] and Spot [26]. We compute the candidate cause according to Sec. 5.1 with pysat [49], using Glucose 4 [3, 65], building on Minisat [65]. We ran experiments on a MacBook Pro with a GHz DualCore Intel Core i7 processor and GB RAM^{3}^{3}3Our prototype implementation and the experimental data are both available at: https://github.com/reactivesystems/explaininghyperpropertyviolations.
Experimental results.
Instance  time()  time()  

Running example  
Security in & out  
Drone example 1  
Drone example 2  
Asymmetric arbiter ’19  see Appendix LABEL:app:candidates  
Asymmetric arbiter  see Appendix LABEL:app:candidates 
The results of our experimental evaluation can be found in Tab. 1. We report on the size of the analyzed counterexample , the size of the violated formula , how long it took to compute the first, overapproximated cause (see time()) and state the approximation itself, the number of computed minimal causes and the time it took to compute all of them (see time()). The Running Example is described in Sec. 3, the instance Security in & out refers to a system which leaks high security input by not satisfying a noninterference property, the Drone examples consider a leaderfollower drone scenario, and the Asymmetric Arbiter instances refer to arbiter implementations that do not satisfy a symmetry constraint. Specifications can be found in App. LABEL:app:specs.
Our first observation is that the cause candidate can be efficiently computed thanks to the iterative computation of unsatisfiable cores (Sec. 5.1). The cause candidate provides a tight overapproximation of possible minimal causes. As expected, the runtime for finding minimal causes increases for larger counterexamples. However, as our experiments show, the overhead is manageable, because we optimize the search for all minimal causes by only considering every subset in instead of naively going over every combination of input events (see Prop. 2). Compared to the computationally heavy task of model checking to get a counterexample, our approach incurs little additional cost, which matches our theoretical results (see Prop. 1). During our experiments, we have found that computing the candidate
first has, additionally to providing a powerful heuristic, another benefit: Even when the computation of minimal causes becomes increasingly expensive,
can serve as an intermediate result for the user. By filtering for important inputs, such as high security inputs, already gives great insight to why the property was violated. In the asymmetric arbiter instance, for example, the input events and of , which cause the violation, immediately catch the eye (c.f App. LABEL:app:candidates).Comparison to HyperVis. HyperVis [47] is a tool for visualizing counterexamples returned from the HyperLTL model checker MCHyper [34]. It highlights the events in the trace that it considers responsible for the violation based on the formula and the set of traces, without considering the system model. However, violations of many relevant security policies such as observational determinism are not caused by events whose atomic propositions appear in the formula, as can be seen in our running example (see Sec. 3 and Ex. 2). When running the highlight function of HyperVis for the counterexample traces on Running example, the output events and will be highlighted, neglecting the decisive high security input . Using our method additionally reveals the input events and