Formal verification, the process of algorithmically generating correctness certificates for a design, and falsification, the process of algorithmically finding trajectories and inputs that lead to a violation of specifications are important steps before a safety-critical control system can be deployed (fainekos2012verification, ; annpureddy2011s, ; sankaranarayanan2012falsification, ). An alternative to these approaches, when a control design is not available but a plant model and specifications are available, is to synthesize a controller that, by construction, guarantees that the specifications are satisfied by the closed-loop system (ozay2017guest, ). The key insight of this paper is to combine ideas from falsification and control synthesis to evaluate control designs for safety.
Consider the problem of evaluating a control design for an autonomous vehicle for safety. What would be a meaningful specification to run a falsification engine against in this case? The hard safety constraint – “do not crash!” – is easy to specify but can be trivially falsified. For instance, if a lead car, with very low speed, cuts in front of the autonomous car traveling with a relatively high speed, a crash is unavoidable. To get “interesting” corner cases, one might constrain the distance at which the lead car cuts in or the speed the lead car is traveling at when it cuts in. But can we systematically generate such constraints/assumptions? If a falsifying trajectory is found, can we say anything about existence of a controller that would be able to steer the vehicle to safety, or is safety simply an impossible task in this situation?
Motivated by these questions, in this paper we propose to use controlled invariant sets (blanchini1999survey, ) to generate interesting corner cases for falsification. By an interesting corner case, we mean initial conditions from which ensuring safety is hard but not necessarily impossible. We restrict our attention to piecewise affine control systems subject to external disturbances (e.g., behavior of the other cars on the road, road profile) and safety constraints given as unions of polyhedra. We propose a scheme to sample initial conditions from the boundary of the invariant set. We also consider the problem of searching for falsifying disturbances (in addition to initial conditions). To this effect, we compute the winning set of a dual game, where control inputs are treated as disturbances and disturbances are treated as control, and where the goal is to reach the unsafe set. The dual strategy obtained by solving the dual game can be used to generate falsifying inputs when the state is within the winning set of the dual game. Greedy heuristics that aim to push the states to the dual game winning set are also proposed.
As an additional advantage, in case a control design is found unsafe using the proposed method, we can supervise this unsafe controller with the controlled invariant set in order to guarantee safety while still using the unsafe controller, which may have favorable performance related properties (nilsson2016correct, ). This supervision idea is similar to the simplex architecture (bak2011sandboxing, ; seto1998dynamic, ), where a performance controller is used together with a simpler controller that has a certified safety envelope and that overwrites the performance controller only when its actions risk safety.
We demonstrate the proposed approach using two autonomous driving functions: adaptive cruise control and lane keeping. Adaptive cruise control aims to regulate the longitudinal dynamics of a vehicle either to a desired speed or a desired headway to a lead vehicle. Lane keeping controls the lateral dynamics of a vehicle to track the center line of the lane. We present safety specifications for both functions. We then apply the proposed approach to a set of controllers, including Comma AI software, an open source autonomous driving package, to reveal potential corner cases leading to specification violation.
2. Main Ingredients
Our goal in this paper is to search for interesting corner cases for falsification of closed-loop control systems. By interesting corner case, we mean a pair of initial condition and external (disturbance) input signal that leads to a trajectory violating a given safety specification, together with a certificate that it is possible to satisfy the specification for this initial condition and external input; therefore violation is indeed avoidable. We summarize the proposed framework in Figures 1 and 2 before detailing the different components.
2.1. Controlled Invariant Sets
Invariance properties are the most basic safety properties where the goal is to avoid an unsafe set at all times, and has been widely studied in the literature (blanchini1999survey, ). The maximal (robust) controlled invariant set is the set of all states inside the safe set from which there exists a controller that can guarantee remaining safe for all future times (under all possible realizations of uncertainty and disturbances).
Formally, we define a controlled invariant set for a continuous-time system using a tangent cone. Let be a set in
; a vectoris called a feasible direction of set at if there exists such that for all . The tangent cone of a set at is then defined to be . Consider a dynamical system described by the following differential equation
where is the state, is the control, and is the disturbance. Here, , , and represent the set of possible states, controls, and disturbances, respectively. Set is called controlled invariant under the dynamics in Eq. (1) if (blanchini1999survey, )
where represents the boundary of set . Set invariance can be defined similarly for discrete-time control systems of the form
A set is controlled invariant under dynamics in Eq. (3) if
For simple linear system dynamics subject to additive disturbance or polytopic uncertainty, it is possible to approximate the maximal invariant set to an arbitrary precision (DeSantis:2004et, ; rungger2017computing, ). In this paper, we used polytopic invariant sets, as is done in (smith2016interdependence, ; nilsson2016correct, ; pn_thesis2017, ). It is also possible to compute controlled invariant sets represented via barrier functions using sum-of-squares optimization (prajna2004safety, ) or approximate them via abstraction-based techniques (roy2011pessoa, ). Invariant set computation can be seen as a safety game between the control input and the disturbance , where the maximal controlled invariant set corresponds to the winning set (i.e., the set of all the initial states from which can enforce safety irrespective of the values of ) in the game for the control input.
The controlled invariant set can be used to supervise a legacy controller to avoid violation of the safety constraint (nilsson2016correct, ), even when a controller for which a safety violation is found in the falsification step is used. The idea is to provide a recursive guarantee on safety by enforcing the trajectory to stay within a controlled invariant set contained by the safe set. The supervisor is a set-valued map that maps the current state to a set of control inputs . Under any control , the next state stays within set under all disturbance. The supervisor overrides the legacy controller in a minimally intrusive way. That is, the supervisor is active and provides a control input in set , whenever the legacy controller gives a control input outside at state . As shown in Fig. 2, when the legacy controller’s input is in , we have the supervisor output .
To be specific, the set can be constructed in the following way. Let be the discrete-time dynamics. We define the set
Given the current state , set is obtained by fixing the component of the points in to be , i.e., . In particular, under the assumption that is a polyhedron (or a union of polyhedra, resp.), is linear in , , , and is a polyhedron, then can also be represented as a polyhedron (or a union of polyhedra, resp.).
2.2. Sampling of the Boundary
We sample the boundary of the controlled invariant set to obtain potentially interesting initial conditions. As mentioned before, the focus in this paper is on controlled invariant sets that can be represented as a finite union of polyhedra. Figure 3 illustrates the boundary sampling scheme of a polyhedron-union set. The gray shaded area is the union of the polyhedra. We assume the union set is contained within a hyper-rectangular domain, and sample along the first dimensions of the domain. These samples corresponds to the red dots in the figure. Then the red dots are projected onto the boundary of the invariant set, which are marked by the blue circles in the figure. In particular, this projection can be done by the following procedure.
We first project each red dot onto the boundary of each polyhedron in the collection. To be specific, we fix the first coordinates of points inside polyhedron to be the same as the red dot . This gives 1 dimensional polyhedron
We then compute the vertex representation of the set using MPT3 (herceg2013multi, ).
The vertices of are admitted if they are not in the interior of other polyhedra.
Remark 1 ().
The proposed sampling scheme can be extended to the case where the controlled invariant set is represented as the union of convex sets in form of . Similar to step 1) in the above procedure, set is created as
Set is a 1-D interval whose bounds can be computed by solving 1-D convex optimization problems and . Also note that a union of convex sets is not necessarily convex and may contain holes. Our proposed approach is able to sample the boundary of the holes as well.
For other types of sets, there is a brief survey in (ghosh2013nearly, ) on the existing approaches to sample the surface of nonconvex polyhedra. Other methods for generating (asymptotically) uniform samples on a polytope’s boundary include the shake-and-bake method (smith1984pointsuniform, ; boender1991shakeandbake, ), and sweep plane method (Leydold98asweep-plane, ), and these can be used as alternatives to the approach described above.
2.3. Computing the Falsifying Inputs
2.3.1. Dual Game
A falsifying scenario consists of two parts: an initial condition and a disturbance input profile. In this part, we show how to compute a falsifying input profile, through solving the so called dual game, given that the initial condition is outside the maximal invariant set. Theoretically, if the initial state is already outside the maximal invariant set, there exists a disturbance input profile that steers the trajectory outside the safe set. However, if the disturbance profile is not selected carefully, it does not necessarily lead to falsification.
We first define some terminology. Let the system dynamics be given by Eq. (1), and let be the safe set we want to stay inside for all time. The invariance game aims at finding the largest controlled invariant set . Figure 4 shows the objective of its dual game: we want to find set , and a dual strategy , under which the states starting from is steered into unsafe set in finite time, as long as .
We solve the dual game by computing the backwards reachable set of unsafe set . For linear discrete-time dynamics, assuming that unsafe set is a polytope, the backwards reachable set can be computed as a collection of polytopes using the same approach in (nilsson2016correct, ). The only difference is that we are now “controlling” the disturbance and trying to be robust to the real control action . To be specific, let the dynamics be
where , , are polytopes. We first compute a sequence of polytopes, starting with , as follows:
We then project each polytope onto space to obtain , and the winning set of the dual game is given by . To determine the dual game strategy at the current state , we locate in one of the projected polytopes , and the dual strategy can be generated by picking such that .
When is nonconvex but can be expressed as a union of polytopes, we compute the backwards reachable set for each polytope and take the union of the obtained backwards reachable sets. This gives a conservative, yet sound, winning set for the dual game. Note that, when the invariant set or the winning set for the dual game is computed via such a conservative approach, there will be a gap between and in Fig. 4, corresponding to a set of initial conditions for which concluding whether they are “interesting” or not is not possible with the computed sets.
2.3.2. Ellipsoid Method
Note that the dual strategy is defined on and is not applicable everywhere on . Thus we need a complementary strategy to generate falsifying inputs for states . Next, we propose some heuristics for computing a complementary strategy. Assume that the safe set is given as a union of polyhedra and denotes the convex-hull of . It is shown in (john2014extremum, ) that, for any compact set , there exists a unique minimum volume ellipsoid (called LJ-ellipsoid of ) covering it. Denote the LJ-ellipsoid of with , which is defined as
where the positive definite matrix parametrizing the ellipsoid can be computed using (rimon1992efficient, ). Define the level of as
It is reasonable to assume that points lying on the higher levels are closer to the unsafe set; hence, driving the system to higher levels would force it either to the unsafe set or to the winning set of the dual game . With this intuition, complementary strategy is defined such that it steers the system to the highest possible level set at each step:
where we assume that control input is known.111When the invariant set is unbounded, LJ-ellipsoid does not exist. In this case can be computed by computing inputs that steer the state closer to by directly minimizing the distance to though the corresponding optimization problem can be more complex. Alternatively, if the rays corresponding to unbounded directions are known, an ellipsoid that is significantly elongated along those directions can be chosen by bounding those rays at a large enough level.
3. System model and specifications
For the case studies we consider two autonomous driving subsystems: adaptive cruise control and lane keeping. An adaptive cruise controller (ACC) controls the speed of the vehicle to follow a desired speed if there is no car in front, and to follow the lead vehicle within some safe following distance (headway) if there is a relatively slower lead vehicle in front. A lane keeping (LK) controller controls the steering of the vehicle to avoid lane departures. Therefore, adaptive cruise control controls the longitudinal dynamics and lane keeping control deals with the lateral dynamics. In the rest of this section, we provide dynamical models used in our examples and formalize safety specifications for both systems.
3.1. Longitudinal and Lateral Dynamics
We use the following model from (nilsson2016correct, ) to describe the longitudinal dynamics of the vehicle:
The system states consist of the following car velocity , lead car velocity , and the headway (i.e., the relative distance between the lead and following car). Control input represents the net force acting on the mass of the following car. The lead car acceleration can be viewed as a disturbance to the system. Finally, constants , , and are parameters of the model. The values of these parameters and the bounds of the variables can be found in Table 1. In particular, the domain the dynamics are defined on is .
|car+cargo mass||1462 ()|
|friction/drag term||51 ()|
|friction/drag term||1.2567 ()|
|friction/drag term||0.4342 ()|
|minimal car velocity||()|
|maximal car velocity||25 ()|
|desired car velocity||20 ()|
|minimal force, comfort||()|
|maximal force, comfort||2870.6 ()|
|minimal force, physical||()|
|minimal force, physical||()|
|maximal acceleration||0.65 ()|
|minimal time headway||1.7 ()|
|minimal headway||4 ()|
The lateral dynamics are described by
where the states are: lateral deviation from the center of the lane (), the lateral velocity (), the yaw-angle deviation in road-fixed coordinates (), and the yaw rate (), respectively. The input is the steering angle of the front wheels, which is limited to lie within and ; and is the desired yaw rate, which we interpret as a time-varying external disturbance and computed from road curvature by where is the (signed) radius of the road curvature and is the vehicle’s longitudinal velocity. Other parameters include , the total mass of the vehicle, and , , , and , which are vehicle geometry and tire parameters. All values can be found in Table 2. Accordingly, the domain the dynamics are defined on is .
|nominal velocity||20 ()|
|car+cargo mass||1462 ()|
car moment of inertia
|vehicle geometry parameter||1.08 ()|
|vehicle geometry parameter||1.62 ()|
|tire parameter||85400 ()|
|tire parameter||90000 ()|
|maximum lateral deviation||0.9 (m)|
|maximum lateral velocity||1 (m/s)|
|maximum yaw-angle deviation||0.15 (rad)|
|maximum yaw rate||0.27 (rad/s)|
|minimum steering angle||(rad)|
|maximum steering angle||0.26 (rad)|
3.2. Formal Specifications for ACC and LK
For ACC, we focus on the safety aspect of requirement in this work. The (safety part of) ISO Standard requirements for Adaptive Cruise Control Systems (isoACCstandard, ) state:
the control input should stay within specified bounds all the times.
whenever the lead car is close in the sense that the headway , the time headway needs to satisfy at all times.
We extract the safety part of the above ISO requirement and express it formally in logic. Define sets
Set is the set of states where the lead car is close, set is the safe set of states, and set contains the allowable control inputs. Adding the speed limits encoded by the domain , the overall specification can be expressed as
To check safety in the presence of a close enough lead car, we assume the states are in and consider the following safety specification, denoted by :
In the later falsification experiments, we will consider violations of different aspects of the specification , that is,
These three safety specifications correspond to small time headway, small distance headway, and crash, respectively. Note that specification implies as . Here we distinguish specification from because violating is considered to be more severe.
For LK, as mandated by the width of roads in the United States (approx. m) and typical car widths (approx. m), the specification states that the car must stay within meters of the center of the lane, i.e. . We also require the other states to remain in the domain as larger values of these states are either physically less meaningful (e.g., can correspond to the vehicle navigating in the reverse direction) or violate passenger comfort requirements. Moreover, the lateral dynamics model we use is valid for relatively smaller ranges of yaw rate, yaw angle and lateral velocity. With these requirements, the overall specification for LK, denoted by , is formally stated as:
Note that state being out of bound should be considered to be a significant safety violation, while the other three states in being out of bounds leads to a less comfortable ride. Therefore, we will independently count the violations of the specification below in the falsification experiments:
4. Case studies
In this section, we evaluate the proposed approach on case studies with different controllers for adaptive cruise control and lane keeping.
In what follows, both the wheel force and the steering angle are bounded quantities. Thus, for controllers that cannot handle such input constraints, we use a saturation function before feeding their output to the system. The saturation function is defined as follows:
In addition to sampling at the boundary of the invariant set, we also sample in the interior of the invariant set to generate less “trickier” initial conditions. These interior points are obtained by shifting (for ACC) or scaling (for LK) the boundary samples.
4.1. Adaptive Cruise Control Results
We computed a controlled invariant set for the longitudinal dynamics in (13), and sampled the boundary of this set with the proposed approach to find falsifying initial conditions. The disturbance profile is computed by (i) solving the dual game, (ii) a simple heuristic that corresponds to the lead car doing a maximum braking, or (iii) the lead car trying to achieve . We explored the following three classes of controllers for meeting the ACC requirement.
For the first controller, we performed feedback linearization followed by pole placement with a hybrid proportional (P) controller, defined as:
where is the proportional gain, the part takes care of the two different ACC modes and given the input saturations.
We also consider hybrid proportional-integral (PI) controllers with the following dynamics:
where is the error state and is the integral coefficient. Similarly, the control input needs to be saturated to obtain practical .
We also designed an MPC controller with a linearized discrete-time model with a sampling period of s using the following formulation:
where , , are the initial conditions and is the length of the prediction horizon. Since the objective contains term , the MPC is hybrid in its nature. To simplify the computation load, we replace the target velocity throughout the predicting horizon by , so that the MPC problem can be solved by a QP solver.
Tables 4-7 summarize the falsification rates for the samples both from the interior and the boundary of the controlled invariant set , with disturbance profile generated by multiple methods. It should be noted that the same controlled invariant set is used to generate initial states to investigate the three different aspects of the safety specification in Eq. (18). This is because set is synthesized against the overall safety specification in Eq. (17). Overall, the MPC controller seems better than the naively designed P controller in terms of safety.
Another key observation is that the falsification rates of the test cases with interior initial conditions can be higher or lower than those on the boundary of set , depending on how the disturbance (i.e., ) profile is generated. In particular, the test cases with interior initial conditions have higher falsification rate than the boundary cases in Table 4-6, and usually have lower falsification rate in Table 7. The key difference between Table 4-6 and Table 7 is that the leading car is usually decelerating (or maintaining speed constant) in the test cases from Table 4-6, while it is accelerating under the test cases from Table 7. In what follows we briefly discuss how this difference affects the falsification rates by the interior initial conditions and by the boundary ones.
When the lead car is decelerating, the dynamics tend to have a “steady state” outside the controlled invariant set . This is true because the lead car’s deceleration shortens the headway and hence pushes the state towards the boundary of . In this case, the falsifications are due to the long term behavior of the dynamics as a trajectory may eventually leave Since such undesired behaviors occur in a longer term, starting from the interior of set may not prevent ultimate falsification.
Moreover, the trajectories initiating from the interior tend to move to the “trickier” parts of the boundary, where safe actions are limited, which increases the falsification rate. We next explain why this is so. First note that a point in the interior of usually has larger relative headway and larger lead car velocity . Consequently, the target velocity defined by has a higher chance to be equal to when starting from the interior. The controller hence accelerates to achieve and maintains the velocity there. Now since the lead car velocity is low (due to deceleration or small initial value), such acceleration will eventually lead to small headway , which will change the target velocity from to . At that moment, however, the following car velocity may be already relatively high. This hence leads to a harder scenario and increases the chance of falsification, which explains the result in Table 4-6.
On the contrary, when lead car’s steady state speed is , i.e. it is mostly accelerating, the dynamics tend to have a “steady state” inside the set . This is true because the lead car’s acceleration enlarges headway and pushes the state towards inside of . In this case, the falsifications are mainly due to the transient state of the dynamics because the state will eventually converge to that steady state inside . By our conjecture, the initial conditions on the boundary of have higher chances for capturing the falsifications due to transient state. This explains why the falsification rate in Table 7 agrees with our conjecture.
To summarize, initial conditions on the boundary help identify safety violations in the transient behavior, whereas input generation techniques tend to capture safety violations due to persistent disturbances (i.e., “steady state”).
4.2. Lane Keeping Results
Let us denote the state vector in Eq. (14) as . We computed a controlled invariant set for the lane keeping model in (14), sampled the boundary of this set with the proposed approach to find falsifying initial conditions, and use various input generation methods to generate road profiles. We explored three classes of controllers for meeting the lane keeping requirement:
A proportional (P) state feedback controller, defined as
Several P controllers are designed by placing the poles in different locations. The control input is obtained by saturating to account for the practical limit of the actuator, i.e., .
A proportional-integral (PI) controller, defined as
where expands to include an error state :
Several PI controllers are designed by choosing different pole locations. Similarly, the control input is obtained by saturating accordingly.
An MPC controller with the following formulation:
Since the input saturation is accounted for by the constraints in the MPC formulation, control input . where is the initial condition, and .
Table 8 summarizes the falsification rates for the above three controllers. The initial conditions are generated by sampling the interior and the boundary of set , and the disturbance profiles are generated using ellipsoid method plus dual game and using a heuristic described as follows:
where is the sampling time of the discrete-time system.
Overall, our MPC design seems safer than the PI design, which is safer than the P controller. Note that none of these designs are tuned properly, the goal is just to demonstrate how controlled invariant sets can be used to evaluate different designs.
|ACC||Proportional controller (P gain)|
|PI controller (P/I gains)||, = 200|
|LK||State feedback (poles)||[-0.93; 0.92; 0.9; 0.8]|
|[0.003; 0.660.34i; 0.4]|
|State feedback w/ integral action (poles)||[-0.93; 0.92; 0.9; 0.8; 0.7]|
|[-0.60.1i; 0.650.2i; 0.7]|
|[0.002; 0.60.4i; 0.4; 0.7]|
|No input generation||Heuristics by Eq. (30)||Ellipsoid method + dual game|
|No input generation||0.15||0.00||0.00||0.15||0.42||0.04||0.00||0.44|
|Heuristic (max brake)||0.15||0.29||0.00||0.29||0.39||0.43||0.00||0.48|
|Ellipsoid method + dual game||0.15||0.17||0.00||0.29||0.36||0.19||0.00||0.48|
|aLead = K(vLead -v_des)||0.43||0.00||0.00||0.43||0.56||0.01||0.00||0.56|
|Comma AI||Comma AI* (without modification)|
|No input generation||0.002||0.868||0.034||0.968||0||1||0.014||1|
|Heuristic by Eq. (30)||0.000||0.880||0.014||0.982||0.272||0.956||0.264||0.996|
|Ellipsoid method + dual game||0.006||0.952||0.038||0.994||1||1||1||1|
4.3. Comma AI
Our framework is flexible enough to evaluate any type of controller as long as their inputs and outputs match the inputs and outputs of the system models used. We can also directly use the source code of a controller after developing a proper interface. In this section, we demonstrate our framework on an open source real-world autonomous driving package developed by Comma AI, a start-up working on self-driving car technologies (see https://comma.ai/). However, since we are just using a simplified model for the vehicle dynamics, the interface might not accurately reflect the performance of the software on an actual car. This can be improved by improving the models and interfaces, but the goal in this section is to simply show the applicability of the framework on realistic control software rather than accurately mimicking the performance.
We describe how we interfaced the Comma AI code (commit 5524dc8222After we settled on a version of Comma AI to use for this project, newer versions of the Comma AI code have changed the lane-keeping module from using a PI to an MPC-based controller. Testing this new controller within our framework is the subject of future work. at https://github.com/commaai) with our ACC and LK framework. The Comma AI code is written in Python. We call the Python code directly from within Matlab by developing appropriate wrappers for input/output matching as described next.
The ACC module of Comma AI outputs two values: gas and brake commands, each normalized to . We scale these gas and brake commands by the physical gas and brake limits of average mid-sized sedans, and , respectively. We then clip the scaled gas and brake commands to the comfort bounds and , respectively. The sum of the scaled gas and brake commands is used as the control input to our system.
The LK module of Comma AI requires extra interfacing with our simulations. Firstly, Comma AI outputs a control between , and we assume that these bounds map linearly onto the range of steering angles . Secondly, Comma AI takes as input the road profile, at discretization, for the upcoming 50 m, which is measured along the tangent line of the current car configuration. To provide this at time step ,
We compute a sequence of future road curvature disturbances , where using one of the input generation methods. To be consistent with prior road profiles, we fix and only compute from scratch. In principle, we can compute entirely from scratch; this setting can be interpreted as driving with Comma AI vision sensor failure/noise, leading to inconsistent roads from prior time steps. However, by ensuring that we provide consistent roads, we give Comma AI the advantage here by assuming that the vision data is exact.
Assuming that the trajectory was obtained from measurements taken at a rate of of the vehicle traveling at