nn-dependability-kit: Engineering Neural Networks for Safety-Critical Systems

1 Introduction

In recent years, neural networks have been widely adapted in engineering automated driving systems with examples in perception [5, 19, 32], decision making [25, 35, 2], or even end-to-end scenarios [3, 40]. As these systems are safety-critical in nature, problems during operation such as failed identification of pedestrians may contribute to risk behaviors. Importantly, the root cause of these undesired behaviors can be independent of hardware faults or software programming errors but can solely reside in the data-driven engineering process

, e.g., unexpected results of function extrapolation between correctly classified training data.

In this paper, we present nn-dependability-kit, an open-source toolbox to support data-driven engineering of neural networks for safety-critical domains. The goal is to provide evidence of uncertainty reduction in key phases of the product life cycle, ranging from data collection, training & validation, testing & generalization, to operation. nn-dependability-kit is built upon our previous work[10, 9, 7, 8, 6], where (a) novel dependability metrics [9, 7] are introduced to indicate uncertainties being reduced in the engineering life cycle, (b) formal reasoning engine [10, 6]

is used to ensure that the generalization does not lead to undesired behaviors, and (c) runtime neuron activation pattern monitoring

[8] is applied to reason whether a decision of a neural network in operation time is supported by prior similarities in the training data.

Figure 1: Using a simplified GSN to understand solutions (Sn) provided by nn-dependability-kit, including the associated goals (G), assumptions (A) and strategies (S).

Concerning related work, our results [10, 9, 7, 8, 6] are within recent research efforts from the software engineering and formal methods community targeting to provide provable guarantee over neural networks [39, 20, 30, 10, 23, 14, 18, 13, 12] or to test neural networks [36, 37, 7, 29, 15, 26]. The static analysis engine inside nn-dependability-kit for formally analyzing neural networks, as introduced in our work in 2017 [10] as a pre-processing step before exact constraint solving, has been further extended to support the octagon abstract domain [28] in addition to using the interval domain. One deficiency of the above mentioned works is how to connect safety goals or uncertainty identification [17, 22, 33] to concrete evidences required in the safety engineering process. This gap is tightened by our earlier work of dependability metrics [9] being partly integrated inside nn-dependability-kit. Lastly, our runtime monitoring technique [8]

is different from known results that either use additional semantic embedding (and integrate it in the loss function) for computing difference measures

[27] or use Monte-Carlo dropout [16] as ensembles: our approach provides a sound guarantee of the similarity measure based on the neuron word distance to the training data.

2 Features and Connections to Safety Cases

Fig. 1 provides a simplified Goal Structuring Notation (GSN) [21] diagram to assist understanding how features provided by nn-dependability-kit contribute to the overall safety goal¹¹1Note that the GSN in Fig. 1 may not be complete, but it can serve as a basis for further extensions.. Our proposed metrics, unless explicitly specified, are based on extensions of our early work [9]. Starting with the goal of having a neural network to function correctly (G1), based on assumptions where no software and hardware fault appears (A1, A2), the strategy (S1) is to ensure that within different phases of the product life cycle, correctness is ensured. These phases include data preparation (G2), training and validation (G3), testing and generalization (G4), and operation (G5).

(Data preparation): Data preparation includes data collection and labeling. Apart from correctly labeling the data (G6), one needs to ensure that the collected data covers all operating scenarios (G7). An artificial example of violating such a principle is to only collect data in sunny weather, while the vehicle is also expected to be operated in snowy weather. Quantitative projection coverage metric [7] and its associated test case generation techniques (Sn1, Sn2), based on the concept of combinatorial testing [24, 11], are used to provide a relative form of completeness against the combinatorial explosion of scenarios.
(Training and validation): Currently, the goal of training correctness is refined into two subgoals of understanding the decision of the neural network (G10) and correctness rate (G8), which is further refined by considering the performance under different operating scenarios (G9). Under the assumption where the neural network is used for vision-based object detection (A5), metrics such as interpretation precision (Sn5) and sensitivity of occlusion (Sn6) are provided by nn-dependability-kit.
(Testing and generalization): Apart from classical performance measures (Sn7), we also test the generalization subject to known perturbations such as haze or Gaussian noise (G11) using the perturbation loss metric (Sn8). Provided that domain knowledge can be formally specified (A3), one can also apply formal verification (Sn9) to examine if the neural network demonstrates correct behavior with respect to the specification (G12).
(Operation): In operation time, as the ground truth is no longer available, a dependable system shall raise warnings when a decision of the neural network is not supported by prior similarities in training (S3). nn-dependability-kit provides runtime monitoring (Sn10) [8] by first using binary decision diagrams (BDD) [4]
to record binarized neuron activation patterns in training time, followed by checking if an activation pattern produced during operation is contained in the BDD.

3 Using nn-dependability-kit

Example 1: Formal Verification of a Highway Front Car Selection Network.

The first example is to formally verify properties of a neural network that selects the target vehicle for an adaptive cruise control (ACC) system to follow. The overall pipeline is illustrated in Fig. 2, where two modules use images of a front facing camera of a vehicle to (i) detect other vehicles as bounding boxes (vehicle detection via YOLO [31]) and (ii) identify the ego-lane boundaries (ego-lane detection). Outputs of these two modules are fed into the third module called target vehicle selection, which is as a neural-network based classifier that reports either the index of the bounding box where the target vehicle is located, or a special class for “no target vehicle”.

Figure 2: Illustration of the Target vehicle selection pipeline for ACC. The input features of the Target vehicle selection NN are defined as follows: 1-8 (possibly up to 10) are bounding boxes of detected vehicles, E is an empty input slot, i.e., there are less than ten vehicles, and L stands for the ego-lane information.

As the target vehicle selection neural network takes a fixed number of bounding boxes, one undesired situation of the neural network appears when the network outputs the existence of a target vehicle with index $i$ , but the corresponding $i$ -th input does not contain a vehicle bounding box (marked with “E” in Fig. 2). For the snapshot in Fig. 2, the neural network should not output class $9$ or $10$ as there are only $8$ vehicle bounding boxes. This undesired property, encoded in the form of linear constraints over inputs and outputs of the neural network, can be proven with the static analysis engine of nn-dependability-kit in seconds, using the below code snippet.

from nndependability.formal import staticanalysis ... staticanalysis.verify(inputMinBound, inputMaxBound, net, True, inputConstraints, riskProperty)

Example 2: Perturbation Loss over German Traffic Sign Recognition Network.

Another example is to analyze a neural network trained under the German Traffic Sign Recognition Benchmark [34] with the goal of classifying various traffic signs. With nn-dependability-kit, one can apply the perturbation loss metric using the below code snippet, in order to understand the robustness of the network subject to known perturbations.

from nndependability.metrics import PerturbationLoss metric = PerturbationLoss.Perturbation_Loss_Metric() ... metric.addInputs(net, image, label) ... metric.printMetricQuantity("AVERAGE_LOSS")

Figure 3: Using nn-dependability-kit to perturb the image of a traffic sign (a) and compute the maximum and average performance drop, by applying perturbation on the data set (b).

As shown in the center of Fig. 3-a, the original image of “end of no overtaking zone” is perturbed by nn-dependability-kit using seven methods. The application of Gaussian noise changes the result of classification, where the confidence of being “end of no overtaking zone” has dropped $83.4 %$ (from the originally identified $100 %$ to $16.6 %$ ). When applying perturbation systematically over the complete data set, the result is summarized in Fig. 3

-b, where one concludes that the network is robust against haze and fog (the maximum probability reduction is close to

0 %

) but should be improved against FGSM attacks [38] and snow.

References

[1]
[2] M. Al-Qizwini, I. Barjasteh, H. Al-Qassab, and H. Radha. Deep learning algorithm for autonomous driving using googlenet. In: IV, pages 89–96, 2017.
[3] M. Bojarski, D. D. Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zieba. End to end learning for self-driving cars. arXiv:1604.07316, 2016.
[4] R. E. Bryant. Symbolic boolean manipulation with ordered binary-decision diagrams. ACM Computing Surveys, 24(3):293–318, 1992.
[5] C. Chen, A. Seff, A. Kornhauser, and J. Xiao. Deepdriving: Learning affordance for direct perception in autonomous driving. In: ICCV, pages 2722–2730, 2015.
[6] C.-H. Cheng, F. Diehl, G. Hinz, Y. Hamza, G. Nührenberg, M. Rickert, H. Ruess, and M. Truong-Le. Neural networks for safety-critical applications - challenges, experiments and perspectives. In: DATE, pages 1005–1006, 2018.
[7] C.-H. Cheng, C.-H. Huang, and H. Yasuoka. Quantitative projection coverage for testing ml-enabled autonomous systems. In: ATVA, pages 126–142, 2018.
[8] C.-H. Cheng, G. Nührenberg, and H. Yasuoka. Runtime monitoring neuron activation patterns. In: DATE, 2019 (to appear).
[9] C.-H. Cheng, G. Nührenberg, C.-H. Huang, and H. Yasuoka. Towards dependability metrics for neural networks. In: MEMOCODE, 2018.
[10] C.-H. Cheng, G. Nührenberg, and H. Ruess. Maximum resilience of artificial neural networks. In: ATVA, pages 251–268, 2017.
[11] C. J. Colbourn. Combinatorial aspects of covering arrays. Le Matematiche, 59(1, 2):125–172, 2004.
[12] S. Dutta, S. Jha, S. Sankaranarayanan, and A. Tiwari. Output range analysis for deep feedforward neural networks. In: NFM, pages 121–138, 2018.
[13] K. Dvijotham, R. Stanforth, S. Gowal, T. Mann, and P. Kohli. A dual approach to scalable verification of deep networks. arXiv:1803.06567, 2018.
[14] R. Ehlers.
Formal verification of piece-wise linear feed-forward neural networks.
In: ATVA, pages 269–286, 2017.
[15] D. J. Fremont, X. Yue, T. Dreossi, S. Ghosh, A. L. Sangiovanni-Vincentelli, and S. A. Seshia. Scenic: Language-based scene generation. arXiv:1809.09310, 2018.
[16] Y. Gal and Z. Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In: ICML, pages 1050–1059, 2016.
[17] L. Gauerhof, P. Munk, and S. Burton.
Structuring validation targets of a machine learning function applied to automated driving.
In: SAFECOMP, pages 45–58, 2018.
[18] T. Gehr, M. Mirman, D. Drachsler-Cohen, P. Tsankov, S. Chaudhuri, and M. T. Vechev. AI2: safety and robustness certification of neural networks with abstract interpretation. In: Oakland, pages 3–18, 2018.
[19] B. Huval, T. Wang, S. Tandon, J. Kiske, W. Song, J. Pazhayampallil, M. Andriluka, P. Rajpurkar, T. Migimatsu, R. Cheng-Yue, et al. An empirical evaluation of deep learning on highway driving. arXiv:1504.01716, 2015.
[20] G. Katz, C. W. Barrett, D. L. Dill, K. Julian, and M. J. Kochenderfer. Reluplex: An efficient SMT solver for verifying deep neural networks. In: CAV, pages 97–117, 2017.
[21] T. Kelly and R. Weaver. The goal structuring notation–a safety argument notation. In DSN 2004 Workshop on Assurance Cases, 2004.
[22] M. Kläs and A. M. Vollmer. Uncertainty in machine learning applications: A practice-driven classification of uncertainty. In: WAISE@SAFECOMP, pages 431–438, 2018.
[23] J. Z. Kolter and E. Wong. Provable defenses against adversarial examples via the convex outer adversarial polytope. arXiv:1711.00851, 2017.
[24] J. Lawrence, R. N. Kacker, Y. Lei, D. R. Kuhn, and M. Forbes. A survey of binary covering arrays. the electronic journal of combinatorics, 18(1):84, 2011.
[25] D. Lenz, F. Diehl, M. Troung Le, and A. Knoll. Deep neural networks for markovian interactive scene prediction in highway scenarios. In: IV. IEEE, 2017.
[26] L. Ma, F. Xu, F. Zhang, J. Sun, M. Xue, B. Li, C. Chen, T. Su, L. Li, Y. Liu, J. Zhao, and Y. Wang. DeepGauge: multi-granularity testing criteria for deep learning systems. In: ASE pages 120–131, ACM.
[27] A. Mandelbaum and D. Weinshall. Distance-based confidence score for neural network classifiers. arXiv:1709.09844, 2017.
[28] A. Miné. The octagon abstract domain. Higher-order and symbolic computation, 19(1):31–100, 2006.
[29] K. Pei, Y. Cao, J. Yang, and S. Jana. Deepxplore: Automated whitebox testing of deep learning systems. In: SOSP, pages 1–18. ACM, 2017.
[30] L. Pulina and A. Tacchella. An abstraction-refinement approach to verification of artificial neural networks. In: CAV, pages 243–257. Springer, 2010.
[31] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. You only look once: Unified, real-time object detection. In: CPVR, pages 779–788, 2016.
[32] A. Sauer, N. Savinov, and A. Geiger. Conditional affordance learning for driving in urban environments. arXiv:1806.06498, 2018.
[33] S. Shafaei, S. Kugele, M. H. Osman, and A. Knoll. Uncertainty in machine learning: A safety perspective on autonomous driving. In WAISE@SAFECOMP, pages 458–464, 2018.
[34] J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In IJCNN, pages 1453–1460. IEEE, 2011.
[35] L. Sun, C. Peng, W. Zhan, and M. Tomizuka. A fast integrated planning and control framework for autonomous driving via imitation learning. arXiv:1707.02515, 2017.
[36] Y. Sun, X. Huang, and D. Kroening. Testing deep neural networks. arXiv:1803.04792, 2018.
[37] Y. Sun, M. Wu, W. Ruan, X. Huang, M. Kwiatkowska, and D. Kroening. Concolic testing for deep neural networks. arXiv:1805.00089, 2018.
[38] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. arXiv:1312.6199, 2013.
[39] T.-W. Weng, H. Zhang, H. Chen, Z. Song, C.-J. Hsieh, D. Boning, I. S. Dhillon, and L. Daniel. Towards fast computation of certified robustness for ReLU networks. arXiv:1804.09699, 2018.
[40] H. Xu, Y. Gao, F. Yu, and T. Darrell. End-to-end learning of driving models from large-scale video datasets. arXiv:1612.01079, 2017.

nn-dependability-kit: Engineering Neural Networks for Safety-Critical Systems

Authors

Testing and verification of neural-network-based safety-critical control software: A systematic literature review

Runtime Monitoring Neural Activation Patterns

Runtime Monitoring Neuron Activation Patterns

DNN or k-NN: That is the Generalize vs. Memorize Question

Entropy-Based Modeling for Estimating Soft Errors Impact on Binarized Neural Network Inference

Towards Repairing Neural Networks Correctly

Proceedings 2nd International Workshop on Causal Reasoning for Embedded and safety-critical Systems Technologies

Code Repositories