Log In Sign Up

Reliability Analysis of Artificial Intelligence Systems Using Recurrent Events Data from Autonomous Vehicles

by   Yili Hong, et al.

Artificial intelligence (AI) systems have become increasingly common and the trend will continue. Examples of AI systems include autonomous vehicles (AV), computer vision, natural language processing, and AI medical experts. To allow for safe and effective deployment of AI systems, the reliability of such systems needs to be assessed. Traditionally, reliability assessment is based on reliability test data and the subsequent statistical modeling and analysis. The availability of reliability data for AI systems, however, is limited because such data are typically sensitive and proprietary. The California Department of Motor Vehicles (DMV) oversees and regulates an AV testing program, in which many AV manufacturers are conducting AV road tests. Manufacturers participating in the program are required to report recurrent disengagement events to California DMV. This information is being made available to the public. In this paper, we use recurrent disengagement events as a representation of the reliability of the AI system in AV, and propose a statistical framework for modeling and analyzing the recurrent events data from AV driving tests. We use traditional parametric models in software reliability and propose a new nonparametric model based on monotonic splines to describe the event process. We develop inference procedures for selecting the best models, quantifying uncertainty, and testing heterogeneity in the event process. We then analyze the recurrent events data from four AV manufacturers, and make inferences on the reliability of the AI systems in AV. We also describe how the proposed analysis can be applied to assess the reliability of other AI systems.


page 1

page 2

page 3

page 4


Statistical Perspectives on Reliability of Artificial Intelligence Systems

Artificial intelligence (AI) systems have become increasingly popular in...

Disengagement Cause-and-Effect Relationships Extraction Using an NLP Pipeline

The advancement in machine learning and artificial intelligence is promo...

Software Reliability Growth Models Predict Autonomous Vehicle Disengagement Events

The acceptance of autonomous vehicles is dependent on the rigorous asses...

Towards safe, explainable, and regulated autonomous driving

There has been growing interest in the development and deployment of aut...

Statistical Analysis of Modern Reliability Data

Traditional reliability analysis has been using time to event data, degr...

Assessing the Safety and Reliability of Autonomous Vehicles from Road Testing

There is an urgent societal need to assess whether autonomous vehicles (...

Artificial Intelligence in Software Testing : Impact, Problems, Challenges and Prospect

Artificial Intelligence (AI) is making a significant impact in multiple ...