Certifiable Deep Importance Sampling for Rare-Event Simulation of Black-Box Systems

11/03/2021
by   Mansur Arief, et al.
0

Rare-event simulation techniques, such as importance sampling (IS), constitute powerful tools to speed up challenging estimation of rare catastrophic events. These techniques often leverage the knowledge and analysis on underlying system structures to endow desirable efficiency guarantees. However, black-box problems, especially those arising from recent safety-critical applications of AI-driven physical systems, can fundamentally undermine their efficiency guarantees and lead to dangerous under-estimation without diagnostically detected. We propose a framework called Deep Probabilistic Accelerated Evaluation (Deep-PrAE) to design statistically guaranteed IS, by converting black-box samplers that are versatile but could lack guarantees, into one with what we call a relaxed efficiency certificate that allows accurate estimation of bounds on the rare-event probability. We present the theory of Deep-PrAE that combines the dominating point concept with rare-event set learning via deep neural network classifiers, and demonstrate its effectiveness in numerical examples including the safety-testing of intelligent driving algorithms.

READ FULL TEXT

page 10

page 24

page 26

research
06/28/2020

Deep Probabilistic Accelerated Evaluation: A Certifiable Rare-Event Simulation Methodology for Black-Box Autonomy

Evaluating the reliability of intelligent physical systems against rare ...
research
10/10/2020

Rare-Event Simulation for Neural Network and Random Forest Predictors

We study rare-event simulation for a class of problems where the target ...
research
04/28/2023

Ensuring Reliable Robot Task Performance through Probabilistic Rare-Event Verification and Synthesis

Providing guarantees on the safe operation of robots against edge cases ...
research
06/19/2021

Accelerated Policy Evaluation: Learning Adversarial Environments with Adaptive Importance Sampling

The evaluation of rare but high-stakes events remains one of the main di...
research
02/14/2021

Achieving Efficiency in Black Box Simulation of Distribution Tails with Self-structuring Importance Samplers

Motivated by the increasing adoption of models which facilitate greater ...
research
12/08/2019

Efficient Black-box Assessment of Autonomous Vehicle Safety

While autonomous vehicle (AV) technology has shown substantial progress,...
research
09/20/2022

Testing Rare Downstream Safety Violations via Upstream Adaptive Sampling of Perception Error Models

Testing black-box perceptual-control systems in simulation faces two dif...

Please sign up or login with your details

Forgot password? Click here to reset