Cyber-physical systems (CPSs) such as aircraft control systems, robots, or partially/fully self-driving cars are very exciting [Nerode, 2007, Pappas, 2011, Alur, 2015]. They are also formidably challenging to get right due to their physical dynamics and all their subtle interactions with the partially unknown environment. That is where differential dynamic logic [Platzer, 2008, Platzer, 2010b, Platzer, 2012c, Platzer, 2012b, Platzer, 2017a, Platzer, 2018a] comes in as a language to express formal specifications for hybrid system models of CPSs. Hybrid systems [Grossman et al., 1993] are the basis of all CPS models, because they feature both discrete dynamics (e.g., coming from the stepwise computation and decisions in a computer) and continuous dynamics (e.g., coming from the physical motion of a car).
Differential dynamic logic provides modal formulas that express that all behavior of hybrid system is such that the states reaches satisfy , which is immediately useful for specification purposes. For example, the formula expresses that: if formula is true initially, then after all runs of hybrid system is formula true. This particular formula is like a Hoare triple except that it works for hybrid systems instead of conventional discrete programs [Hoare, 1969]. Yet, differential dynamic logic is more general, because it also makes it possible to state logical formulas of other forms beyond safety. The formula of differential dynamic logic plays a similar role to the formula of temporal logic [Pnueli, 1977, Prior, 1957] in speaking about always being true, except that it is parametrized by the particular hybrid systems after all whose runs is true. That makes it possible for formulas to refer to multiple hybrid systems at once. The formula , e.g., expresses that if after all runs of is true, then also after all runs of is true. The use of differential dynamic logic is not limited to specification purposes but also provides verification techniques by systematically reducing formulas to simpler formulas by logical transformations until the resulting formulas can be proved.
Fortunately, differential dynamic logic is not just for pen-and-paper proofs but is implemented111The prover is available at http://keymaeraX.org/ in the theorem prover [Fulton et al., 2015] that makes it possible to conduct rigorous formal proofs with tool support and substantial levels of proof automation including invariant generation [Sogokon et al., 2019]. The tool was instrumental for verifying challenging applications, including ground robot obstacle avoidance [Mitsch et al., 2017b], railway systems [Mitsch et al., 2017a], and the Next-generation Airborne Collision Avoidance System ACAS X [Jeannin et al., 2017]. Safety guarantees provided by differential dynamic logic can also transcend models and translate into reality. implements the transfer of safety proofs for CPS models to CPS implementations by synthesizing provably correct runtime monitors with ModelPlex [Mitsch and Platzer, 2016b], which result in CPS executables that are formally verified in a chain of theorem provers [Bohrer et al., 2018]
. ModelPlex is also the basis for enabling safe artificial intelligence in cyber-physical systems[Platzer, 2019a]
by wrapping reinforcement learning in a verified safety sandbox[Fulton and Platzer, 2018] and steering back toward safety when outside the confounds of a well-modeled part of the system behavior [Fulton and Platzer, 2019]
. These results that will be surveyed here provide rigorous safety guarantees for CPS implementations (even those that involve machine learning) without having to deal with the entire complexity of their implementation during verification.
While a brief overview is given here, significantly more elaborate detail can be found in a recent textbook222Including supporting slides and video lectures are at http://lfcps.org/lfcps/ on Logical Foundations of Cyber-Physical Systems [Platzer, 2018a] as well as a research monograph on Logical Analysis of Hybrid Systems [Platzer, 2010b]. A technical survey of basic differential dynamic logic can be found in a LICS’12 tutorial [Platzer, 2012c].
2 CPS Programs
Hybrid systems are mathematical models that combine discrete dynamical systems with continuous dynamical systems as is crucial to understand CPSs. A variety of different types of hybrid system models have been developed, including hybrid automata [Nerode and Kohn, 1992, Alur et al., 1992], hybrid Petri nets [David and Alla, 2001], hybrid CSP [Chaochen et al., 1995] and hybrid process algebras [Cuijpers and Reniers, 2005]. The Logical Foundations of Cyber-Physical Systems are best built on hybrid programs [Platzer, 2008, Platzer, 2010b, Platzer, 2012c, Platzer, 2012b, Platzer, 2017a, Platzer, 2018a], which represent hybrid systems in a programming language that includes differential equations. The relationship of hybrid programs to hybrid automata is similar to that of regular languages to finite automata. The advantage of hybrid programs for verification purposes is that they make it possible to leverage programming language compositionality principles to completely decompose the analysis of hybrid programs into respective analyses of their pieces [Platzer, 2008, Platzer, 2012b, Platzer, 2017a].
Unlike conventional discrete programs, hybrid programs provide differential equations as primitives for continuous dynamics. In order to do justice to the demands of cyber-physical systems, hybrid programs feature real number computations and natively offer nondeterminism with which uncertainty about the evolution of the physical world can be expressed. Such nondeterminism is crucial to accurately capture the fact that we have limited knowledge about what other agents in the physical environment may do (it really is hard to know what exactly the car in front of you will do when). But nondeterminism also comes in handy when we want to be deliberately imprecise about what happens when in our own control in order to arrive at a more abstract but easier model (even if we could in principle know exactly when our car controller speeds up or slows down as a function of distance, speed limit, gear, gas-air-ratio, engine rotations per minute, and passenger comfort goals, we may be better off retaining a more abstract nondeterministic decision to either accelerate or brake).
Definition 1 (Hybrid programs).
The syntax of hybrid programs (HP) is described by the following grammar where are hybrid programs, is a variable and are terms, is a (usually first-order) logical formula:
Polynomial terms suffice, but rational functions can be used (when not dividing by zero [Bohrer et al., 2019]). The assignment instantaneously changes the value of variable to the value of the term by a discrete jump. The test checks if formula is true in the current state and aborts the execution discarding the run otherwise. That is, tests have absolutely no effect on the state of the system except that they abort executions that do not pass them. Such tests are useful to impose conditions on the (especially branching) execution of a system. Most importantly, the differential equation can be followed for any nondeterministic amount of time (including 0), but only as long as the logical formula is true at every time along the solution of . In particular, just like in a failed test, it is impossible to run a differential equation for any amount of time at all unless is true in the beginning.
The nondeterministic choice can either run the hybrid program or the program . It is the fundamental branching construct, making no prior commitment as to which side is run. Unlike which either runs or , the sequential composition first runs and then . It is the fundamental construct for running things in succession. Finally, the nondeterministic repetition will repeatedly run hybrid program any arbitrary natural number of times, including possibly 0 times. It is the fundamental construct for repeated actions.
Example 1 (Bouncing ball).
As a simple hybrid system, consider a bouncing ball at altitude with velocity that is bouncing up and down over time subject to gravity where is the constant for gravity (fig:bouncingball-simple-trajectory).
Since the bouncing ball is unable to fall through the ground at position 0, the differential equation is limited to remain above ground . That is, the bouncing ball can continue falling in gravity for any amount of time, but can never fall through the cracks in the ground. After the bouncing ball stops its continuous evolution, it follows a nondeterministic choice whose right branch tests whether the ball is in the air (with the test x≠0) in which case no change happens. Its left branch tests if the ball lies on the ground (test x=0) and then inverts the velocity by a discrete assignment () subject to a damping coefficient . That is why the discrete controller in line 2 is can also be written as . Because the velocity will be flipped from negative to positive when on the ground, that will send the ball flying back up in the air as soon as it next continues along after repeating the loop. The repetition operator () at the end of the hybrid program in fig:bouncingball-simple-trajectory indicates that it can be repeated any arbitrary number of times.
Note that all choices in all operators of hybrid programs are resolved nondeterministically. The differential equation can be stopped at any time (just cannot violate the evolution domain constraint ). When stopped above ground (), the subsequent discrete controller has no effect and the only way forward is to, nondeterministically, stop or repeat the loop and, if it the loop is repeated, continue along the same differential equation in line 1. At the latest on the ground (), however, does the differential equation have to stop and give the discrete control a chance to inspect the state and flip the sign of the velocity. Also note how some choices, e.g., the nondeterministic choice in fig:bouncingball-simple-trajectory are nondeterministic but, because of the subsequent tests in both subprograms, really only one choice will succeed in every state. That is why a nondeterministic default is helpful, because more deterministic behavior can always be defined by imposing suitable tests in the execution of a nondeterministic program. Conversely, a deterministic program cannot properly be made nondeterministic anymore.
While bouncing balls are really no cyber-physical systems (they apparently forgot their computing devices), their behavior can still be explained very well with a prototypical hybrid system where the mix of discrete and continuous dynamics comes from describing the sudden bounce back from the ground versus the continuous flying and falling above ground. It is much easier to understand bouncing balls from such a hybrid systems perspective than to investigate the fast-paced continuous deformation of the ball on the ground happening at a very different time scale than its continuous flying and falling in gravity.
What is crucial for cyber-physical systems is not only the addition of differential equations, but also to embrace nondeterminism and real-valued quantities.
The effect of running a hybrid program is that the state of the system changes, which is why the semantics of hybrid programs is defined as the relation it induces between initial and final states. Because of the nondeterminism, some hybrid programs may reach multiple (or no) final states from a given initial state. A state is a mapping that assigns a real number to each variable . The set of all states is denoted . The real value of a term in state is written and defined as usual in real arithmetic.
Definition 2 (Transition semantics).
The semantics of an HP defines the transition relation  ⊆Σ ×Σ of initial and final states inductively:
where x=x ∧, iff for all times 0≤≤r: x=x ∧ with [state](x) [t][state][state=φ(t)](x)() and except at .
That is, the relation x=x ∧ holds when is a solution of the differential equation that respects the evolution domain constraint at every point in time and, of course, no other variable changes except those indicated by the differential equation. The semantics is explicit change: nothing changes unless the hybrid program indicates how.
The semantics of a nondeterministic choice is the union of the individual semantics’, which is where the notation comes from. The semantics of a sequential composition is the relation composition . The semantics of a nondeterministic repetition is the reflexive transitive closure of the semantics of or, equivalently, the semantics of all the -fold iterations of . The semantics of an assignment changes the value of the variable in the new state to the real value that the right-hand side term had in the old state . The semantics of a test does not change the state but a transition is only possible in a state that is in the set of all states in which is true (written and defined in def:dL-semantics below).
3 Differential Dynamic Logic
Differential dynamic logic [Platzer, 2008, Platzer, 2010b, Platzer, 2012c, Platzer, 2012b, Platzer, 2017a, Platzer, 2018a] provides specification and verification techniques for hybrid systems written as hybrid programs.
The defining constructs of differential dynamic logic are the pair of modalities and for every hybrid program , which can be used in front of any formula . The formula is true in any state in which is true after all runs of hybrid program . The formula is true if there is a run of hybrid program after which is true. The rest of is built like first-order logic of real arithmetic including quantifiers over the reals. Real arithmetic makes the most sense when talking about the real positions and velocities of CPSs.
Definition 3 (formulas).
The syntax of formulas of differential dynamic logic is described by the following grammar where are formulas, are terms, is a variable and is a hybrid program:
Bi-implications are defined syntactically as mutual implication . Likewise, other operators such as are definable as usual.
By combining the operators it is easy to express various correctness statements about hybrid systems. For example, a formula of the shape says that if is true then all runs of satisfy . Hence, is valid (true in all states) when formula is, indeed, true after all runs of hybrid system that start in an initial state where formula is true (safety). Likewise, says that if is true then there is a run of after which is true (liveness). Other correctness properties mix multiple modalities in the same formula. For example, expresses that if is true (initially), then after all behaviors of , there is a run of that makes true in the end. The formula expresses a conditional equivalence that if is true (initially), then all behaviors of satisfy iff all behaviors of satisfy . Quantifiers can be useful, e.g., to say that for all (initial) positions there is an (initial) velocity for which all behavior of car obeys a safety property as .
Example 2 (Bouncing ball conjecture).
Continuing with the hybrid program from ex:bouncing-ball-HP, a formula expressing safety of the bouncing ball is the following:
It expresses that if the bouncing ball starts at the initial height above ground, then it will always remain above ground but below initial height (). Unfortunately, no matter how intuitive such a claim may look like at first glance, the above formula is not actually true for all (initial) values of . Of course, if the initial values of and disagree, then the formula is vacuously true, because the left-hand side of its implication is false. But additional assumptions are also needed on the other variables as well to make the formula true for all (initial) values of all variables. Which assumptions are those?
Strictly speaking, the definitions of hybrid programs (def:HP) and of differential dynamic logic formulas (def:dL-formula) are mutually inductive, because hybrid programs occur in modal formulas while formulas occur in tests or evolution domain constraints. That is why their semantics is also defined by mutual induction (def:HP-transition and def:dL-semantics). But since not much intuition is lost when first imagining only simple arithmetic formulas can occur in hybrid programs (so-called poor test ), this subtlety is often glossed over in the interest of a simple presentation.
The meaning of a formula is the set of all states in which it is true (from which the set of all states in which it is false can be read off by complementation).
Definition 4 (Semantics).
The semantics of formulas is defined inductively, for each formula , as the set of states, written ⊆Σ, in which is true:
A formula is valid, written , iff it is true in all states, i.e., .
The formula is true in any state from which all states reachable after running hybrid program satisfy . The formula is true in any state from which there is a state that is reachable after running hybrid program and that satisfies .
Valid formulas are the most valuable ones, because we can rely on them being true no matter what state a system is presently in. For example, an implication is valid iff , i.e., the set of all states in which is true is a subset of the set of all states in which is true. A bi-implication is valid iff , i.e., the set of all states in which is true is the same as the set of all states in which is true.
Example 3 (Bouncing ball safety).
Augmenting ex:bouncing-ball-conjecture with sufficiently many assumptions on other variables to make the formula about the behavior of bouncing balls valid (so true in all states) leads, e.g., to the following formula:
Of course, slightly more general formulas are valid as well. But assuming damping coefficient , for example, has the pleasant effect of considerably simplifying notation subsequently.
4 Dynamic Axioms for Dynamical Systems
Definitions 2 and 4 give a rigorous mathematical semantics but, except in simple cases, working through their uncountable sets of states does not exactly give a particularly practical way of establishing whether formulas are true in a state let alone valid (so true in all states). Differential dynamic logic is not just useful for specifying CPS but also for verifying CPS thanks to its proof calculus, with which the validity of formulas can be established by a finite syntactic proof instead of the uncountably infinite sets of states in the semantics.
Axioms and proof rules for differential dynamic logic [Platzer, 2012b, Platzer, 2017a] are listed in fig:dL. The usual axioms and proof rules for propositional connectives and first-order logic can be used as a basis [Platzer, 2008, Platzer, 2018a] but are not discussed further here.
The idea behind the axioms in fig:dL is that, when used to (equivalently) reduce the (usually left) blue side to the rest, they symbolically decompose a property of a larger hybrid program into a range of properties of its pieces. For example, the axiom of nondeterministic choice choiceb is an equivalence that makes it possible to replace any formula of the shape equivalently with a corresponding conjunction that expresses safety of smaller subprograms. Since both sides are equivalent according to axiom choiceb, replacing one side by the other in any context does not change the overall truth. But the right-hand side of axiom choiceb consists of structurally simpler formulas, because all its hybrid programs are simpler than those on the left-hand side, so that using axiom choiceb to replace the left by the right side makes progress in simplifying formulas. Depending on the polarity (number of negations in negation normal form) with which occurs in a formula, either the right-to-left or left-to-right implication of axiom choiceb is needed to justify correctness of that transformation. Since axiom choiceb, like most others, is an equivalence axiom, it is easy to see (and easy to prove [Platzer, 2017a]) why any occurrence of a formula of the form can be replaced equivalently by the corresponding equivalent regardless of the context.
The sequential composition axiom composeb performs a more subtle simplifying decomposition by making it possible to equivalently replace formulas of the form with more complex hybrid programs into nested modalities with more complex postconditions but smaller hybrid programs (no anymore).
The assignment axiom assignb has no more subprograms to work with but expresses that is true after assigning iff is true before. Tests are handled by axiom testb which expresses that holds after running test iff holds now, because the test can only run successfully when was indeed true. Axiom evolveb replaces differential equations equivalently by a universally quantified property of their solution satisfying and . This works for simple differential equations whose solutions are expressible in real arithmetic (and continues to work with slight modifications in the presence of evolution domain constraints), but is hopeless for “serious” differential equations.
More involved differential equations use inductive proof techniques based on differential invariants [Platzer, 2010a, Platzer, 2010b, Platzer, 2017a] that can prove all their real arithmetic invariants in differential dynamic logic [Platzer and Tan, 2018]. The basic idea of differential invariants is to prove them inductively by proving that their derivatives along the differential equation satisfy the same logical relations. Other steps in such proofs successively accumulate knowledge about the evolution of the differential equation in its evolution domain constraint (differential cut) or soundly edit the differential equation (differential ghost) as elaborated elsewhere [Platzer, 2012c, Platzer, 2017a, Platzer and Tan, 2018].
4.2 Proof Rules
Loops can be unwound finitely with the iteration axiom iterateb or proved by the induction axiom Ieq. As usual, the postcondition may need to be strengthened to identify a proper loop invariant . This strengthening ultimately uses Kripke’s modal modus ponens axiom K or Gödel’s generalization rule G, but is also packaged up with a derived proof rule in sequent calculus:
As usual in classical logics, the meaning of a sequent is the meaning of the formula . That is, the meaning of a comma in the list of assumptions or antecedent is conjunctive while the meaning of a comma in the succedent is disjunctive. Sequents are particularly useful for normalizing the shape of logical formulas.
Finally, axiom diamond uses dualities of modalities to make duals of all reasoning principles for box modalities from the other axioms available for diamond modalities. The vacuous axiom V is special in that it is only applicable if the free variables of do not intersect the bound variables written by , i.e., . It shows in a single step that the truth of formulas remains unaffected by running if does not depend on any variable that program changes. A more refined uniform substitution calculus for [Platzer, 2017a] completely avoids any such side conditions in the entire calculus. Uniform substitutions make logics significantly easier to implement and improve robustness, which is why the implementation of is entirely based on uniform substitutions [Fulton et al., 2015].
The differential dynamic logic proof calculus successively reduces properties of hybrid systems to formulas of real arithmetic, which is a surprisingly tame logic where validity is decidable [Tarski, 1951, Collins, 1975, Collins and Hong, 1991], although in general of doubly exponential complexity in the number of quantifier block alternations [Weispfenning, 1988, Davenport and Heintz, 1988].
Example 4 (Bouncing ball proof).
The conjectured validity of the bouncing ball formula from ex:bouncing-ball-dL, of course, deserves a proper proof as justification. It is easy to conduct such a proof after choosing a suitable loop invariant . The proof uses abbreviations for the initial condition (), postcondition (), and for the differential equation with its evolution domain constraint ():
The first step of the proof (starting at the desired conclusion at the very bottom) is to use a propositional implication rule implyr to assume the left-hand side of the implication () in the antecedent and proceed to prove its right-hand side. Using rule loop with a loop invariant (identified later), postcondition of the nondeterministic repetition is proved by showing that follows from the initial condition (left premise), inductively remains true when running a round of the loop (middle premise), and finally implies the postcondition (right premise): [array] [loop] A j(x,v) !j(x,v) x..; (x=0;v-cvx≠0)j(x,v) !j(x,v) B(x,v) [implyr] A (x..; (x=0;v-cvx≠0))B(x,v) A (x..; (x=0;v-cvx≠0))B(x,v)
The proof of the middle premise continues in fig:dL-bouncing-ball-proof. That proof is follow-your-nose except that it saves some writing by using a derived monotonicity rule that makes it possible to replace postcondition by a postcondition implying :
The remaining four branches of arithmetic and the differential equation property make it fairly easy to prove or disprove concrete invariant candidate choices for using decision procedures for real arithmetic [Collins and Hong, 1991] and either solution axioms or proofs for ODE invariants [Platzer and Tan, 2018]. It is also not difficult to generate a suitable invariant from the desired ODE invariance property [Platzer, 2018a]:
This invariant makes it possible to complete the bouncing ball proof with the constant assumptions , which are easily added [Platzer, 2018a].
Working with the solution suffices for proving all differential equation properties arising in the above example. But differential invariants enable a significantly easier proof from the differential equation itself instead of its more complicated solution. The following special case of the more general differential invariants proof rule suffices here:
Here, dIeq shows that an equality always remains true after an ODE by proving that its differentials and are equal after assigning to . [array] [dIeq] [Dassignb] [qear] x≥0 2gv=-2v(-g) x≥0 xvv-g2gx=-2vv 2gx=2gH-v^2 x=vv=-gx≥02gx=2gH-v^2
4.3 Soundness and Completeness
While not the focus of this overview, differential dynamic logic also enjoys very strong theoretical properties. The most important property of soundness (all formulas that have a proof are valid) is a conditio sine qua non, a condition without which logic could not be. And, of course, soundness holds for the calculus [Platzer, 2008, Platzer, 2012b, Platzer, 2017a] as has even been cross-verified in both Isabelle/HOL and Coq [Bohrer et al., 2017]. The converse question of completeness, i.e., whether all true formulas have a proof, is more complicated. It is answered in the affirmative both for completeness relative to differential equations and relative to discrete dynamics (precise statements can be found in the literature [Platzer, 2008, Platzer, 2012b, Platzer, 2017a]).
Theorem 1 (Continuous relative completeness of [Platzer, 2008, Platzer, 2012b]).
The calculus is a sound and complete axiomatization of hybrid systems relative to differential equations, i.e., every valid formula can be derived in the calculus from valid differential equation tautologies.
Theorem 2 (Discrete relative completeness of [Platzer, 2012b]).
The calculus is a sound and complete axiomatization of hybrid systems relative to discrete dynamics, i.e., every valid formula can be derived in the calculus from valid discrete system tautologies.
In particular, relative completeness theorems (which have constructive proofs) imply that the required (variants or) invariants are always expressible in to succeed with the proofs. If the invariants are in real arithmetic, the remaining questions about differential equations are completely provable in the calculus (again referring to the literature [Platzer and Tan, 2018] for precise statements):
Theorem 3 (Invariant completeness of [Platzer and Tan, 2018]).
The calculus is a sound and complete axiomatization of arithmetic invariants of differential equations, i.e., all true real arithmetic invariants of differential equations are provable in (and all false ones are disprovable in ).
Moreover, real arithmetic invariants of differential equations are decidable directly by a derived axiom of [Platzer and Tan, 2018]. In particular, in the common case where the respective hybrid system invariants that always exist in by thm:dL-complete for valid safety formulas are actually real arithmetical, thm:dL-invariant-complete guarantees that the resulting differential equation properties are provable in if and only if they are true. In theory, the only thing that can go wrong in a hybrid systems safety proof of a valid formula is that the required invariants exist in but are not real arithmetical. In practice, what can still go wrong, in addition, is that scalability limits make the resulting computations infeasible. This is where user guidance in a theorem prover can help get proofs unstuck with user insights about the CPS design that the proof automation fails to find.
4.4 Proofs in The Theorem Prover
The differential dynamic logic proof calculus is implemented in the theorem prover [Fulton et al., 2015] (along with an implementation of differential game logic for hybrid games [Platzer, 2015] that is not the focus here). The prover implements the uniform substitution calculus for differential dynamic logic [Platzer, 2017a] (and differential game logic [Platzer, 2018b, Platzer, 2019b]). The advantage of a uniform substitution design is that this leads to a minimal soundness-critical core consisting only of a verbatim copy of each axiom formula besides the uniform substitution application mechanism and renaming. For performance reasons, also implements a propositional sequent calculus with Skolemization [Platzer, 2008] to efficiently normalize logical formulas. This gives a minimal soundness-critical core of under 2000 lines of code.
At the same time, provides a language for custom tactics [Fulton et al., 2017], sophisticated proof automation, including invariant generation [Sogokon et al., 2019], and a versatile user interface [Mitsch and Platzer, 2016a], but those are markedly outside the soundness-critical part of even if still just as important for practical verification in .
For deciding formulas of real arithmetic, calls Mathematica [Wolfram, 2019] or Z3 [de Moura and Bjørner, 2008, Jovanović and de Moura, 2012]. Instead of trusting those solvers, polynomial witnesses can be used to rigorously prove the validity of universal real arithmetic formulas [Harrison, 2007, Platzer et al., 2009]. In theory, that is complete for universal real arithmetic but only works in practice for medium complexity problems. While proof-producing real arithmetic decision procedures exist [McLaughlin and Harrison, 2005], they are only used in the old 3 prover [Platzer and Quesel, 2008], because their scalability is limited to low-dimensional problems. In the older 3, real arithmetic can also be decided with virtual substitution [Weispfenning, 1997] using Redlog [Dolzmann and Sturm, 1997], which works in cases where the polynomial degrees remain at most cubic. Generally, however, more scalable verification of nonlinear real arithmetic remains an important future challenge.
5 ModelPlex: Model Safety Transfer
provides automatic and interactive and programmable ways for rigorously proving the safety (or other properties) of hybrid systems in differential dynamic logic. However useful and reassuring that is for the hybrid system and its control, it is important to realize that the outcome is still a statement about the hybrid system, which is a mathematical model of the cyber-physical system. A safety proof for a hybrid system only implies the safety of a cyber-physical system to the extent that the hybrid system is an adequate model of the CPS. If we rename a input file from bouncing-ball.kyx to car.kyx and prove it, then we have not made any actual car any safer, because bouncing ball models are quite inappropriate models for cars. In a sense, of course, the appropriate course of action is that we then need to get a better model and put it into car.kyx. But how do we then really argue that a safety proof of the model in the improved car.kyx file entails that the car is safe while a safety proof of the bouncing-ball.kyx model did not? Or what would it take to argue that bouncing-ball.kyx at least is a good model for bouncing balls (spoiler alert: it is not as long as we keep the assumption ). This requires us to prove that we built the right model (does it suitably model all real behavior) and not just that the model was built right (its controllers work safely in that model).
Proving safety of a real CPS is surprisingly subtle, precisely because it does not help to fall back to even more models along the way that need to be justified, too, to accurately reflect reality. The solution to cut through this Gordian knot of model mysteries is ModelPlex, which provides a way to rigorously transfer safety proofs of hybrid systems models to safety results about real CPS implementations [Mitsch and Platzer, 2016b]. The way it works is that ModelPlex conducts a proof in differential dynamic logic about the safe behavior of the real CPS that combines the safety proof about the hybrid systems model with proofs obtained online from monitors that witness the compatibility of the observed real executions of the CPS with the safety-critical part of the hybrid systems model. This results in a proof in which the majority of the work is performed offline ahead of time, while a few critical measurements performed by monitors fill in online at runtime the parts of the proof that link to reality and are unprovable offline. But the required monitors are proved safe in ahead of time and synthesized along with their correctness proof in from the safety proof of the original formula.
When run on the CPS, synthesized ModelPlex monitors check model compliance and safeguard decisions obtained by the unverified controller implementation (fig:ModelPlex-logo). In that respect, ModelPlex monitors are used like Simplex monitors [Seto et al., 1998], except that ModelPlex monitors are proved correct, synthesized from the verified models, and not only worry about the safety of the controls but also ensure safe compliance with the whole model (hence the name ModelPlex).
Example 5 (Correct controller monitor).
A provably correct runtime monitor for the bouncing ball controller from ex:bouncing-ball-dL is as follows [Platzer, 2018a, Ex. 19.3]:
This formula expresses a relation for acceptable transitions of the controller implementation from old values of and to new values and .
Example 6 (Correct model monitor).
A provably correct runtime monitor for the bouncing ball model from ex:bouncing-ball-dL is the following [Platzer, 2018a, Prop. 19.1]:
Monitors get even more subtle with the uncertainty of partially observable hybrid systems, but ModelPlex generalizes suitably [Mitsch and Platzer, 2018]. The point is, however, that both monitors from Examples 5 and 6 are violated when comparing the behavior with a real bouncing ball, whether uncertainty is taken into account or not. This shows, for example, that is an unrealistic assumption for a true damping coefficient, which, after all, makes a real bouncing ball bounce back up less over time (recall fig:bouncingball-simple-trajectory).
6 VeriPhy: Executable Proof Transfer
The ModelPlex monitors synthesized for safeguarding the transfer of safety properties of a hybrid system to a real CPS are a crucial ingredient. But they are also not directly executable without error on a real CPS. The reason is that ModelPlex monitors are correct mathematical conditions for the safety transfer from models to reality, but are still phrased in real arithmetic. Real arithmetic provides the right setting in which to establish the safety of motion in the real world from one real-valued position to another connected by the real-valued differential equations of the model. But such infinite precision real arithmetic is hardly executable on any actual CPU in any reasonable amount of time (or at all). There is also still a gap between knowing the right formulas to monitor from ModelPlex and being able to run proper safe executables on the computation devices of a CPS.
This is where VeriPhy comes in, the verified pipeline with which verified controller executables can be generated from verified cyber-physical system models [Bohrer et al., 2018]. VeriPhy adds machine executability in provably correct ways to the mathematical accuracy of ModelPlex monitors. It turns a hybrid systems model verified safe in into executable machine code for a sandbox that inherits safety for the real CPS in provably safe ways with a chain of proofs in theorem provers. The resulting CPS sandbox checks and corrects all actions of an unverified CPS controller provided separately, and safely flags plant violations when possibly unsafe deviations are detected compared to the safe behavior of the model. The last step of the VeriPhy pipeline compiles provably correctly to CakeML [Kumar et al., 2014], thanks to which machine code can be generated provably correctly for multiple CPUs, e.g., x64 and arm6 architectures.
At this point, one might take the bouncing ball model from ex:bouncing-ball-dL that was verified in ex:bouncing-ball-dL-proof and generate provably correct machine code for it with VeriPhy, which will use the synthesized ModelPlex controller and plant monitors from Examples 5 and 6, respectively. But, admittedly, purchasing a genuine bouncing ball would have led to an easier and more realistic execution platform, which, furthermore, would not have fallen prey to the deficiencies caused immediately by the unrealistic assumption . Yet, that is a consequence of the fact that the discrete operations in ex:bouncing-ball-dL were just a description of bouncing ball physics, and not within the purview of an ingenious cyber controller that is in active need to control how balls bounce. The discrete control of a robot’s ping-pong paddle to react upon the motion of a ping-pong ball [Platzer, 2018a] would have led to a related model in which VeriPhy would serve a better purpose.
7 Safe Learning in CPS
The logical foundations that have been surveyed thus far give a solid basis for the development of provably correct cyber-physical systems [Platzer, 2018a]. But an additional twist arises for autonomous cyber-physical systems [Platzer, 2019a] that achieve smart and largely unsupervised decisions on their own using machine learning and artificial intelligence (AI) techniques. The extra complication is that their behavior is not just a consequence of their programming, but also of their experience and the data they learned from, which is stored, e.g., in policy tables that influence their future decisions [Russel and Norvig, 1995]. Lauded for their flexibility, this AI phenomenon, however, makes the behavior of AI algorithms much harder to predict, which is adverse to their safety.
Technically, one could embed all operations that the AI performs explicitly into the discrete part of a hybrid program. In practice, however, a rendition of the safety-critical data and code then quickly reaches completely infeasible sizes. Due to its use of Markov Decision Process policy optimization, an explicit rendition of the Next-generation Airborne Collision Avoidance System ACAS X[Kochenderfer et al., 2012], for example, would have led to a hybrid program with about half a trillion cases. That is why ACAS X was verified in by splitting off the policy comparison from a general symbolic verification [Jeannin et al., 2017].
More generally, AI and machine learning can be understood as unverified implementations whose compliance with provably safe operations of the CPS can be ensured using ModelPlex [Mitsch and Platzer, 2016b]. While its primary intent is to provide ways of bridging safety of models to safety of implementations, ModelPlex does not actually mind for what reasons part of the CPS are unverified, including for complexity reasons or due to the use of AI and learning in the CPS.
Sandboxed Execution of a Learned CPS.
Combining reinforcement learning with ModelPlex safety monitors enables provably safe reinforcement learning in CPS [Fulton and Platzer, 2018]. Reinforcement learning repeatedly chooses actions and observes outcomes in the (simulated or real) environment and makes those actions more likely if the outcome was favorable and less likely if it was not [Sutton and Barto, 1998]. The most obvious way of benefiting from a provably safe ModelPlex monitor in a learning CPS is to leave learning alone while training and then, during deployment, simply safeguard all actions performed by the trained CPS using the ModelPlex monitors. The advantage of this approach is that it is conceptually simple, but the downside is that the experience obtained from the learning algorithm will be used in a real environment that differs from the training environment, precisely because the ModelPlex monitors only interfere after deployment not during training, which may lead to a suboptimal performance.
Sandboxed Learning of a CPS.
Another alternative is to integrate reinforcement learning with ModelPlex safety monitors more deeply and use the feedback from safety monitors throughout the learning [Fulton and Platzer, 2018]. First of all will the reinforcement learning algorithm learn to act in the same sandboxed environment that it would otherwise be acting in after deployment. Second of all does that give the learning algorithm a chance to learn the concept of safety and learn which actions will be vetoed due to ModelPlex monitor violations. Furthermore can the reinforcement learning algorithm benefit from the early feedback that the ModelPlex monitors provide, which immediately flag actions as problematic if they could have unsafe downstream consequences, instead of having to wait for a scenario where an actual disaster happened later on, and then find out how to propagate this outcome back to the earlier root cause decision. Such early feedback has been observed to significantly speed up convergence of reinforcement learning. Another advantage of sandboxed learning is that the resulting policy (when started from a safe initial policy or fallback) can be provably safe [Fulton and Platzer, 2018]. A downside of sandboxed learning is that the learning may become overly reliant on the sandbox interference, and that it cannot learn behavior outside the known safe sandbox. This can be compensated for by allowing exploration outside known safe actions (within a simulated training environment).
Recovery Learning of a CPS.
Yet another possibility for combining formal methods with machine learning is to continue to use information from safety proofs even outside known safe parts of the world [Fulton and Platzer, 2018]. In that case do quantitative versions of ModelPlex monitors serve as reward signals that tend to pull the system back from unsafe into safe space. Beyond the experimental observation that this enables a safe recovery outside well-modeled parts of the world is it possible to give rigorous safety proofs of the resulting behavior of the learning CPS for the case of multiple possible models that are not all wrong or that can be modified to safely fit reality with verification-preserving model updates [Fulton and Platzer, 2019]. The basic idea is to use the conjunction of all ModelPlex monitors of plausible models to determine which action is safe while discarding models whose predictions did not end up happening. Satisfying assignments to logical combinations of ModelPlex monitors of different models can be used to actively plan differentiating experiments and converge a.s. to the true model.
Of course, all three of these types of modifications of learning algorithms or deployments of learned systems could be used with any other monitor, but then the crucial ingredient of provable safety of the outcome would be missing.
There is no point in using any learning to find out how to act in the bouncing ball model from ex:bouncing-ball-dL, because its discrete part is a deterministic if-then-else statement allowing no room for any actual decisions. But suppose the ground is replaced by a ping-pong paddle that can actively hit the ball harder than a passive floor could. Then, there suddenly is a control choice to be made and it can make sense to use learning from experience to find out how hard to best hit the ball under what circumstance. This would require a change in the model replacing the fixed damping coefficient by a variable pushback from the ground as a function of how quickly the ping-pong paddle swings.
While short and succinct out of necessity, the purpose of this article was to survey the Logical Foundations of Cyber-Physical Systems approach, which is detailed in a recent textbook of the same name [Platzer, 2018a]. This approach provides a solid foundation for the analysis and correct development of cyber-physical systems and their quite subtle controllers. The theorem prover provides automatic and interactive and programmable ways for rigorously proving the safety (or other properties) of hybrid systems in differential dynamic logic. Extensions were summarized that transfer safe models to a safe reality, provide provably safe execution on CPS, or augment safe CPS with safe learning to obtain the best of both worlds. Extensions from hybrid systems to hybrid games are also implemented in the theorem prover , but presented along with their generalizations elsewhere [Platzer, 2015, Platzer, 2017b, Platzer, 2018a]. Extensions from hybrid systems to distributed hybrid systems [Platzer, 2012a] are fascinating and useful, but presently only implemented in previous-generation theorem provers [Platzer and Quesel, 2008, Renshaw et al., 2011].
Acknowledgements. I thank Brandon Bohrer for his feedback on this overview.
- [Alur, 2015] Alur, R. (2015). Principles of Cyber-Physical Systems. MIT Press, Cambridge.
- [Alur et al., 1992] Alur, R., Courcoubetis, C., Henzinger, T. A., and Ho, P.-H. (1992). Hybrid automata: An algorithmic approach to the specification and verification of hybrid systems. In [Grossman et al., 1993], pages 209–229.
- [Bohrer et al., 2019] Bohrer, B., Fernandez, M., and Platzer, A. (2019). dL: Definite descriptions in differential dynamic logic. In [Fontaine, 2019], pages 94–110.
- [Bohrer et al., 2017] Bohrer, B., Rahli, V., Vukotic, I., Völp, M., and Platzer, A. (2017). Formally verified differential dynamic logic. In Bertot, Y. and Vafeiadis, V., editors, Certified Programs and Proofs - 6th ACM SIGPLAN Conference, CPP 2017, Paris, France, January 16-17, 2017, pages 208–221, New York. ACM.
- [Bohrer et al., 2018] Bohrer, B., Tan, Y. K., Mitsch, S., Myreen, M. O., and Platzer, A. (2018). VeriPhy: Verified controller executables from verified cyber-physical system models. In Grossman, D., editor, Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2018, pages 617–630. ACM.
- [Chaochen et al., 1995] Chaochen, Z., Ji, W., and Ravn, A. P. (1995). A formal description of hybrid systems. In Alur, R., Henzinger, T. A., and Sontag, E. D., editors, Hybrid Systems, volume 1066 of LNCS, pages 511–530, Berlin. Springer.
- [Collins, 1975] Collins, G. E. (1975). Quantifier elimination for real closed fields by cylindrical algebraic decomposition. In Barkhage, H., editor, Automata Theory and Formal Languages, volume 33 of LNCS, pages 134–183, Berlin. Springer.
- [Collins and Hong, 1991] Collins, G. E. and Hong, H. (1991). Partial cylindrical algebraic decomposition for quantifier elimination. J. Symb. Comput., 12(3):299–328.
- [Cuijpers and Reniers, 2005] Cuijpers, P. J. L. and Reniers, M. A. (2005). Hybrid process algebra. J. Log. Algebr. Program., 62(2):191–245.
- [Davenport and Heintz, 1988] Davenport, J. H. and Heintz, J. (1988). Real quantifier elimination is doubly exponential. J. Symb. Comput., 5(1/2):29–35.
- [David and Alla, 2001] David, R. and Alla, H. (2001). On hybrid Petri nets. Discrete Event Dynamic Systems, 11(1-2):9–40.
- [de Moura and Bjørner, 2008] de Moura, L. M. and Bjørner, N. (2008). Z3: an efficient SMT solver. In Ramakrishnan, C. R. and Rehof, J., editors, Tools and Algorithms for the Construction and Analysis of Systems, 14th International Conference, TACAS 2008, Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2008, Budapest, Hungary, March 29-April 6, 2008. Proceedings, volume 4963 of LNCS, pages 337–340. Springer.
- [Dolzmann and Sturm, 1997] Dolzmann, A. and Sturm, T. (1997). Redlog: Computer algebra meets computer logic. ACM SIGSAM Bull., 31(2):2–9.
- [Fontaine, 2019] Fontaine, P., editor (2019). International Conference on Automated Deduction, CADE’19, Natal, Brazil, Proceedings, volume 11716 of LNCS. Springer.
- [Fulton et al., 2017] Fulton, N., Mitsch, S., Bohrer, B., and Platzer, A. (2017). Bellerophon: Tactical theorem proving for hybrid systems. In Ayala-Rincón, M. and Muñoz, C. A., editors, ITP, volume 10499 of LNCS, pages 207–224. Springer.
- [Fulton et al., 2015] Fulton, N., Mitsch, S., Quesel, J.-D., Völp, M., and Platzer, A. (2015). KeYmaera X: An axiomatic tactical theorem prover for hybrid systems. In Felty, A. and Middeldorp, A., editors, CADE, volume 9195 of LNCS, pages 527–538, Berlin. Springer.
- [Fulton and Platzer, 2018] Fulton, N. and Platzer, A. (2018). Safe reinforcement learning via formal methods: Toward safe control through proof and learning. In McIlraith, S. A. and Weinberger, K. Q., editors, AAAI, pages 6485–6492. AAAI Press.
- [Fulton and Platzer, 2019] Fulton, N. and Platzer, A. (2019). Verifiably safe off-model reinforcement learning. In Vojnar, T. and Zhang, L., editors, TACAS, Part I, volume 11427 of LNCS, pages 413–430. Springer.
- [Grossman et al., 1993] Grossman, R. L., Nerode, A., Ravn, A. P., and Rischel, H., editors (1993). Hybrid Systems, volume 736 of LNCS, Berlin. Springer.
- [Harrison, 2007] Harrison, J. (2007). Verifying nonlinear real formulas via sums of squares. In Schneider, K. and Brandt, J., editors, TPHOLs, volume 4732 of LNCS, pages 102–118, Berlin. Springer.
- [Hoare, 1969] Hoare, C. A. R. (1969). An axiomatic basis for computer programming. Commun. ACM, 12(10):576–580.
- [Jeannin et al., 2017] Jeannin, J., Ghorbal, K., Kouskoulas, Y., Schmidt, A., Gardner, R., Mitsch, S., and Platzer, A. (2017). A formally verified hybrid system for safe advisories in the next-generation airborne collision avoidance system. STTT, 19(6):717–741.
- [Jovanović and de Moura, 2012] Jovanović, D. and de Moura, L. M. (2012). Solving non-linear arithmetic. In Gramlich, B., Miller, D., and Sattler, U., editors, Automated Reasoning - 6th International Joint Conference, IJCAR 2012, Manchester, UK, June 26-29, 2012. Proceedings, volume 7364 of LNCS, pages 339–354. Springer.
- [Kochenderfer et al., 2012] Kochenderfer, M. J., Holland, J. E., and Chryssanthacopoulos, J. P. (2012). Next-generation airborne collision avoidance system. Lincoln Laboratory Journal, 19(1):17–33.
- [Kumar et al., 2014] Kumar, R., Myreen, M. O., Norrish, M., and Owens, S. (2014). CakeML: a verified implementation of ML. In Jagannathan, S. and Sewell, P., editors, The 41st Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL ’14, San Diego, CA, USA, January 20-21, 2014, pages 179–192. ACM.
- [LICS, 2012] LICS (2012). Logic in Computer Science (LICS), 2012 27th Annual IEEE Symposium on, Los Alamitos. IEEE.
- [McLaughlin and Harrison, 2005] McLaughlin, S. and Harrison, J. (2005). A proof-producing decision procedure for real arithmetic. In Nieuwenhuis, R., editor, CADE, volume 3632 of LNCS, pages 295–314. Springer.
- [Mitsch et al., 2017a] Mitsch, S., Gario, M., Budnik, C. J., Golm, M., and Platzer, A. (2017a). Formal verification of train control with air pressure brakes. In Fantechi, A., Lecomte, T., and Romanovsky, A., editors, RSSRail, volume 10598 of LNCS, pages 173–191. Springer.
- [Mitsch et al., 2017b] Mitsch, S., Ghorbal, K., Vogelbacher, D., and Platzer, A. (2017b). Formal verification of obstacle avoidance and navigation of ground robots. I. J. Robotics Res., 36(12):1312–1340.
- [Mitsch and Platzer, 2016a] Mitsch, S. and Platzer, A. (2016a). The KeYmaera X proof IDE: Concepts on usability in hybrid systems theorem proving. In Dubois, C., Masci, P., and Méry, D., editors, 3rd Workshop on Formal Integrated Development Environment, volume 240 of EPTCS, pages 67–81.
- [Mitsch and Platzer, 2016b] Mitsch, S. and Platzer, A. (2016b). ModelPlex: Verified runtime validation of verified cyber-physical system models. Form. Methods Syst. Des., 49(1-2):33–74. Special issue of selected papers from RV’14.
- [Mitsch and Platzer, 2018] Mitsch, S. and Platzer, A. (2018). Verified runtime validation for partially observable hybrid systems. CoRR, abs/1811.06502.
- [Nerode, 2007] Nerode, A. (2007). Logic and control. In Cooper, S. B., Löwe, B., and Sorbi, A., editors, CiE, volume 4497 of LNCS, pages 585–597, Berlin. Springer.
- [Nerode and Kohn, 1992] Nerode, A. and Kohn, W. (1992). Models for hybrid systems: Automata, topologies, controllability, observability. In [Grossman et al., 1993], pages 317–356.
- [Pappas, 2011] Pappas, G. J. (2011). Wireless control networks: modeling, synthesis, robustness, security. In Caccamo, M., Frazzoli, E., and Grosu, R., editors, Proceedings of the 14th ACM International Conference on Hybrid Systems: Computation and Control, HSCC 2011, Chicago, IL, USA, April 12-14, 2011, pages 1–2, New York. ACM.
- [Platzer, 2008] Platzer, A. (2008). Differential dynamic logic for hybrid systems. J. Autom. Reas., 41(2):143–189.
- [Platzer, 2010a] Platzer, A. (2010a). Differential-algebraic dynamic logic for differential-algebraic programs. J. Log. Comput., 20(1):309–352.
- [Platzer, 2010b] Platzer, A. (2010b). Logical Analysis of Hybrid Systems: Proving Theorems for Complex Dynamics. Springer, Heidelberg.
- [Platzer, 2012a] Platzer, A. (2012a). A complete axiomatization of quantified differential dynamic logic for distributed hybrid systems. Log. Meth. Comput. Sci., 8(4:17):1–44. Special issue for selected papers from CSL’10.
- [Platzer, 2012b] Platzer, A. (2012b). The complete proof theory of hybrid systems. In [LICS, 2012], pages 541–550.
- [Platzer, 2012c] Platzer, A. (2012c). Logics of dynamical systems. In [LICS, 2012], pages 13–24.
- [Platzer, 2015] Platzer, A. (2015). Differential game logic. ACM Trans. Comput. Log., 17(1):1:1–1:51.
- [Platzer, 2017a] Platzer, A. (2017a). A complete uniform substitution calculus for differential dynamic logic. J. Autom. Reas., 59(2):219–265.
- [Platzer, 2017b] Platzer, A. (2017b). Differential hybrid games. ACM Trans. Comput. Log., 18(3):19:1–19:44.
- [Platzer, 2018a] Platzer, A. (2018a). Logical Foundations of Cyber-Physical Systems. Springer, Cham.
- [Platzer, 2018b] Platzer, A. (2018b). Uniform substitution for differential game logic. In Galmiche, D., Schulz, S., and Sebastiani, R., editors, IJCAR, volume 10900 of LNCS, pages 211–227. Springer.
- [Platzer, 2019a] Platzer, A. (2019a). The logical path to autonomous cyber-physical systems. In Parker, D. and Wolf, V., editors, QEST, volume 11785 of LNCS, pages 25–33. Springer.
- [Platzer, 2019b] Platzer, A. (2019b). Uniform substitution at one fell swoop. In [Fontaine, 2019], pages 425–441.
- [Platzer and Quesel, 2008] Platzer, A. and Quesel, J.-D. (2008). KeYmaera: A hybrid theorem prover for hybrid systems. In Armando, A., Baumgartner, P., and Dowek, G., editors, IJCAR, volume 5195 of LNCS, pages 171–178, Berlin. Springer.
- [Platzer et al., 2009] Platzer, A., Quesel, J.-D., and Rümmer, P. (2009). Real world verification. In Schmidt, R. A., editor, CADE, volume 5663 of LNCS, pages 485–501, Berlin. Springer.
- [Platzer and Tan, 2018] Platzer, A. and Tan, Y. K. (2018). Differential equation axiomatization: The impressive power of differential ghosts. In Dawar, A. and Grädel, E., editors, LICS, pages 819–828, New York. ACM.
- [Pnueli, 1977] Pnueli, A. (1977). The temporal logic of programs. In FOCS, pages 46–57. IEEE.
- [Prior, 1957] Prior, A. (1957). Time and Modality. Clarendon Press, Oxford.
- [Renshaw et al., 2011] Renshaw, D. W., Loos, S. M., and Platzer, A. (2011). Distributed theorem proving for distributed hybrid systems. In Qin, S. and Qiu, Z., editors, ICFEM, volume 6991 of LNCS, pages 356–371. Springer.
- [Russel and Norvig, 1995] Russel, S. and Norvig, P. (1995). Artificial Intelligence: a Modern Approach. Prentice-Hall.
- [Seto et al., 1998] Seto, D., Krogh, B., Sha, L., and Chutinan, A. (1998). The Simplex architecture for safe online control system upgrades. In American Control Conference, volume 6, pages 3504–3508.
- [Sogokon et al., 2019] Sogokon, A., Mitsch, S., Tan, Y. K., Cordwell, K., and Platzer, A. (2019). Pegasus: A framework for sound continuous invariant generation. In ter Beek, M., McIver, A., and Oliviera, J. N., editors, FM, volume 11800 of LNCS, pages 138–157. Springer.
- [Sutton and Barto, 1998] Sutton, R. S. and Barto, A. G. (1998). Reinforcement Learning. The MIT Press, Cambridge.
- [Tarski, 1951] Tarski, A. (1951). A Decision Method for Elementary Algebra and Geometry. University of California Press, Berkeley, 2nd edition.
- [Weispfenning, 1988] Weispfenning, V. (1988). The complexity of linear problems in fields. J. Symb. Comput., 5(1-2):3–27.
- [Weispfenning, 1997] Weispfenning, V. (1997). Quantifier elimination for real algebra — the quadratic case and beyond. Appl. Algebra Eng. Commun. Comput., 8(2):85–101.
- [Wolfram, 2019] Wolfram (2019). Mathematica. Wolfram Research, Inc. Champaign, IL.