Assessing Safety-Critical Systems from Operational Testing: A Study on Autonomous Vehicles

by   Xingyu Zhao, et al.

Context: Demonstrating high reliability and safety for safety-critical systems (SCSs) remains a hard problem. Diverse evidence needs to be combined in a rigorous way: in particular, results of operational testing with other evidence from design and verification. Growing use of machine learning in SCSs, by precluding most established methods for gaining assurance, makes operational testing even more important for supporting safety and reliability claims. Objective: We use Autonomous Vehicles (AVs) as a current example to revisit the problem of demonstrating high reliability. AVs are making their debut on public roads: methods for assessing whether an AV is safe enough are urgently needed. We demonstrate how to answer 5 questions that would arise in assessing an AV type, starting with those proposed by a highly-cited study. Method: We apply new theorems extending Conservative Bayesian Inference (CBI), which exploit the rigour of Bayesian methods while reducing the risk of involuntary misuse associated with now-common applications of Bayesian inference; we define additional conditions needed for applying these methods to AVs. Results: Prior knowledge can bring substantial advantages if the AV design allows strong expectations of safety before road testing. We also show how naive attempts at conservative assessment may lead to over-optimism instead; why extrapolating the trend of disengagements is not suitable for safety claims; use of knowledge that an AV has moved to a less stressful environment. Conclusion: While some reliability targets will remain too high to be practically verifiable, CBI removes a major source of doubt: it allows use of prior knowledge without inducing dangerously optimistic biases. For certain ranges of required reliability and prior beliefs, CBI thus supports feasible, sound arguments. Useful conservative claims can be derived from limited prior knowledge.



page 1

page 2

page 3

page 4


Assessing the Safety and Reliability of Autonomous Vehicles from Road Testing

There is an urgent societal need to assess whether autonomous vehicles (...

Reliability Assessment and Safety Arguments for Machine Learning Components in Assuring Learning-Enabled Autonomous Systems

The increasing use of Machine Learning (ML) components embedded in auton...

A Safety Framework for Critical Systems Utilising Deep Neural Networks

Increasingly sophisticated mathematical modelling processes from Machine...

Software Reliability Growth Models Predict Autonomous Vehicle Disengagement Events

The acceptance of autonomous vehicles is dependent on the rigorous asses...

Closed-Loop Policies for Operational Tests of Safety-Critical Systems

Manufacturers of safety-critical systems must make the case that their p...

An Accelerated Approach to Safely and Efficiently Test Pre-produced Autonomous Vehicles on Public Streets

Various automobile and mobility companies, for instance, Ford, Uber, and...

Bootstrapping confidence in future safety based on past safe operation

With autonomous vehicles (AVs), a major concern is the inability to give...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.