Generating Probabilistic Safety Guarantees for Neural Network Controllers

by   Sydney M. Katz, et al.

Neural networks serve as effective controllers in a variety of complex settings due to their ability to represent expressive policies. The complex nature of neural networks, however, makes their output difficult to verify and predict, which limits their use in safety-critical applications. While simulations provide insight into the performance of neural network controllers, they are not enough to guarantee that the controller will perform safely in all scenarios. To address this problem, recent work has focused on formal methods to verify properties of neural network outputs. For neural network controllers, we can use a dynamics model to determine the output properties that must hold for the controller to operate safely. In this work, we develop a method to use the results from neural network verification tools to provide probabilistic safety guarantees on a neural network controller. We develop an adaptive verification approach to efficiently generate an overapproximation of the neural network policy. Next, we modify the traditional formulation of Markov decision process (MDP) model checking to provide guarantees on the overapproximated policy given a stochastic dynamics model. Finally, we incorporate techniques in state abstraction to reduce overapproximation error during the model checking process. We show that our method is able to generate meaningful probabilistic safety guarantees for aircraft collision avoidance neural networks that are loosely inspired by Airborne Collision Avoidance System X (ACAS X), a family of collision avoidance systems that formulates the problem as a partially observable Markov decision process (POMDP).



There are no comments yet.


page 14

page 15

page 16

page 17

page 18

page 20


Verification of Image-based Neural Network Controllers Using Generative Models

Neural networks are often used to process information from image-based s...

Verifying Aircraft Collision Avoidance Neural Networks Through Linear Approximations of Safe Regions

The next generation of aircraft collision avoidance systems frame the pr...

Validation of Image-Based Neural Network Controllers through Adaptive Stress Testing

Neural networks have become state-of-the-art for computer vision problem...

Verification and Control of Turn-Based Probabilistic Real-Time Games

Quantitative verification techniques have been developed for the formal ...

ART: Abstraction Refinement-Guided Training for Provably Correct Neural Networks

Artificial Neural Networks (ANNs) have demonstrated remarkable utility i...

Sampling-Based Robust Control of Autonomous Systems with Non-Gaussian Noise

Controllers for autonomous systems that operate in safety-critical setti...

Fast Neural Network Verification via Shadow Prices

To use neural networks in safety-critical settings it is paramount to pr...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.