Log In Sign Up

Limits of Probabilistic Safety Guarantees when Considering Human Uncertainty

by   Richard Cheng, et al.

When autonomous robots interact with humans, such as during autonomous driving, explicit safety guarantees are crucial in order to avoid potentially life-threatening accidents. Many data-driven methods have explored learning probabilistic bounds over human agents' trajectories (i.e. confidence tubes that contain trajectories with probability δ), which can then be used to guarantee safety with probability 1-δ. However, almost all existing works consider δ≥ 0.001. The purpose of this paper is to argue that (1) in safety-critical applications, it is necessary to provide safety guarantees with δ < 10^-8, and (2) current learning-based methods are ill-equipped to compute accurate confidence bounds at such low δ. Using human driving data (from the highD dataset), as well as synthetically generated data, we show that current uncertainty models use inaccurate distributional assumptions to describe human behavior and/or require infeasible amounts of data to accurately learn confidence bounds for δ≤ 10^-8. These two issues result in unreliable confidence bounds, which can have dangerous implications if deployed on safety-critical systems.


page 3

page 4


Learning-Based Safety-Stability-Driven Control for Safety-Critical Systems under Model Uncertainties

Safety and tracking stability are crucial for safety-critical systems su...

Architectural patterns for handling runtime uncertainty of data-driven models in safety-critical perception

Data-driven models (DDM) based on machine learning and other AI techniqu...

Cooperation for Scalable Supervision of Autonomy in Mixed Traffic

Improvements in autonomy offer the potential for positive outcomes in a ...

Positive Trust Balance for Self-Driving Car Deployment

The crucial decision about when self-driving cars are ready to deploy is...

Learning-based Model Predictive Control for Safe Exploration and Reinforcement Learning

Learning-based methods have been successful in solving complex control t...

Oversight of Unsafe Systems via Dynamic Safety Envelopes

This paper reviews the reasons that Human-in-the-Loop is both critical f...

Guaranteed Conformance of Neurosymbolic Models to Natural Constraints

Deep neural networks have emerged as the workhorse for a large section o...