Neural Networks for Safety-Critical Applications - Challenges, Experiments and Perspectives

09/04/2017 ∙ by Chih-Hong Cheng, et al. ∙ 0

We propose a methodology for designing dependable Artificial Neural Networks (ANN) by extending the concepts of understandability, correctness, and validity that are crucial ingredients in existing certification standards. We apply the concept in a concrete case study in designing a high-way ANN-based motion predictor to guarantee safety properties such as impossibility for the ego vehicle to suggest moving to the right lane if there exists another vehicle on its right.



There are no comments yet.


page 1

page 2

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

The recent burst of applying artificial neural network (ANN) technologies has created an impact on applications such as autonomous driving. Although using ANN-based techniques had shown great promise (e.g., substantially superior image recognition [6]) compared to classical approaches, there have been huge barriers in using neural networks in safety critical domains (e.g., report from NASA [2]).

In this paper, we propose a methodology for enabling the usage of ANN by considering reasonable extensions for existing safety standards (Sec. II). We examine the technology readiness of our proposed methodology by applying a case study regarding highway motion prediction for autonomous driving (Sec. III), and address further research needs (Sec. IV).

Implementation Existing standard Fine-grained specification-to-code traceability
understandability Adaptation for ANN (

) Fine-grained neuron-to-feature traceability

Implementation Existing standard Verification based on testing and classical coverage criteria such as MC/DC
correctness Adaptation for ANN () coverage criteria such as MC/DC
() formal analysis against safety properties
Specification Existing standard Validation via prototyping, design-time analysis, validity and product acceptance test
validity Adaptation for ANN () Validating data as a new type of specification
TABLE I: Extending the concept in certify safety-critical systems to new opportunities brought by neural networks.

Ii Certification Considerations for Dependable Neural Networks

For certification of safety critical systems, safety is established by rigorous engineering processes (i.e., these processes are defined in a way such that engineering complying to these processes can eliminate or prevent errors). Although it is more process-oriented than function-oriented, the basic principle of (1) ensuring that the specification is correct and (2) ensuring that an implementation satisfies the specification is well perceived. Table I summarizes three critical aspects over the underlying intention of certifying safety-critical systems, namely specification validity, implementation understandability, and implementation correctness.

  • The validity of specification is important to ensure that one “builds the right system”. Several methods can be used in this regard, such as prototyping, design-time analysis and reviews, or product acceptance tests.

  • The well-behaving of an implementation is captured by two aspects: (1) understandability via requirement-to-code traceability, and (2) correctness via extensive testing, with coverage criterion such as Modified Condition / Decision Coverage (MC/DC).

Although these approaches are valid for classical engineering using V-models, applying them on neural networks has created the following issues:

  • (Black-box structure

    ) For ANN-based systems, implementations consist of layers of neurons operating on and transforming high-dimensional vectors. This makes understandability arguments such as fine-grained requirement-to-code traceability difficult.

  • (Testing for correctness claims

    ) Depending on the activation function, applying traditional coverage-based approaches makes the system testing either trivially satisfiable or almost intractable. (i) When one uses

    as the activation function, one only needs one test case to satisfy MC/DC as there is no if-then-else branch in every neuron. (ii) When one uses ReLU as the activation function, every neuron contains an if-then-else statement. MC/DC is then intractable, as branching possibilities are exponential to the number of neurons.

  • (Implicit specification) For implementing systems using ANNs, the specification refers to a combination of data (which specifies input-output behaviors) as well as classical specifications for domain knowledge such as traffic or safety rules. The “specification knowledge” inside the data is implicit, compared to cases such as traffic rules.

Based on the above issues, Table I further summarizes our considered additions towards safety certification of ANNs.

  1. [label=()]

  2. (Neuron-to-feature understandability) One should provide confidence regarding the meaning of a neural network by associating individual neurons with conditions (features) when it can be activated.

  3. (From testing to formal analysis) The result of certification should provide (best effort) correctness claims over the (partially incomplete) classical specification, such as obeying traffic rules or ensuring road safety. As testing approaches its limitation, we suggest to apply formal methods such as static analysis or symbolic reasoning.

  4. (Validating the “new specification”) One needs to check the validity of the data, to ensure that only sanitized data will be used in training. For examples such as autonomous driving, one needs to enhance raw data with sure guarantees such as no data containing risky driving has been introduced for training the maneuver of vehicles.

Iii Case Study: Highway Motion Prediction for Autonomous Vehicles

We outline how we applied the strategy above in verifying a highway overtaking ANN-based motion predictor used in autonomous driving (developed by Lenz et al. [7]). Figure 1 provides a snapshot on the simulation of the vehicle.

In Figure 1

, the ANN-based predictor takes three categories of inputs: (i) its own speed profile, (ii) parameters of its nearest surrounding vehicles for each orientation, and (iii) the road condition. The total number of input variables to the network is 84. Given the current state of the perceived environment, it produces in real-time the probability distribution over all possible actions for a vehicle, characterized as a Gaussian mixture model. The action of the ego vehicle is decomposed into two parts: (i) indicator over possible

lateral velocity (i.e., if it is feasible to switch lanes), and (ii) indicator over longitudinal acceleration (i.e., if it is feasible to accelerate). In Figure 1, the motion predictor on the right suggests to slightly decelerate and to switch to left lanes, as the generated Gaussian mixture is within the lower left part.

Fig. 1: Simulation of the vehicle (left) and the switch-lane motion suggested by the neural network (right).

One of the most critical safety requirements is to ensure that if there is a vehicle in the left of the ego vehicle, the predictor never suggests a large left velocity to the ego vehicle; when such a scenario occurs, it may lead to crashes. In this example, it is regulated that the mean value of the probability distribution should be limited to certain threshold.

Once we validated that the training data never contains such inputs (as in Sec. II (A)), we perform formal verification (as in Sec. II (B)) following the methodology developed by Cheng et al. [3], which encodes the structure of a neural network into a set of mixed integer linear constraints. With the technology we are able to successfully verify safety properties. Surprisingly, we have trained a couple of neural networks under the same data, but not all of them can guarantee the safety property (see Fig. II for a summary of verification results, being experimented on a Google VM with 12 Cores).

ANN maximum lateral velocity, when exists a vehicle in the left verification time
0.688497 5.4s
0.467385 549.1s
2.10916 28.2s
1.95859 645.9s
1.72781 13351.2s
n.a. (unable to find maximum) time-out
Prove that the lateral velocity can never be larger than 11059.8s
TABLE II: Results of verifying ANN-based motion predictors.

Iv Concluding Remarks

The proposed certification methodology, during the case study, has also indicated further research needs.

  1. [label=()]

  2. During the study, we found that implementation understandability can only be partially achieved by technologies such as deconvolution [8].

  3. Scalability of automated verification requires improvement (cf. Table II for required verification time). Recent results on quantized neural networks [5] might make verification more scalable via an encoding to bitvector theories in SMT [4].

  4. Apart from verification, another important direction is to consider training under known properties on the target function (known as hints [1]), such as safety rules.


  • [1] Y.S. Abu-Mostafa. Hints. Neural Computation 7(4), pages 639–671, 1995.
  • [2] S. Bhattacharyya, D. Cofer, D Musliner, J. Mueller, and E. Engstrom. Certification considerations for adaptive systems. In: ICUAS, pages 270–279. IEEE, 2015.
  • [3] C.-H. Cheng, G. Nührenberg, and H. Rueß. Maximum resilience of artificial neural networks. In ATVA, 2017.
  • [4] Z3: An efficient SMT solver. L. De Moura, N. Bjørner. Z3: An efficient SMT solver. In TACAS, pages 337–340, 2008.
  • [5] I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, Y. Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. In: arXiv:1609.07061, 2016.
  • [6] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097–1105, 2012.
  • [7] D. Lenz, F. Diehl, M. Troung Le, and A. Knoll. Deep neural networks for markovian interactive scene prediction in highway scenarios. In: IV. IEEE, 2017.
  • [8] M. Zeiler, G. W Taylor, and R. Fergus. Adaptive deconvolutional networks for mid and high level feature learning. In ICCV, pages 2018–2025. IEEE, 2011.