Adaptive Stress Testing without Domain Heuristics using Go-Explore

04/08/2020
by   Mark Koren, et al.
0

Recently, reinforcement learning (RL) has been used as a tool for finding failures in autonomous systems. During execution, the RL agents often rely on some domain-specific heuristic reward to guide them towards finding failures, but constructing such a heuristic may be difficult or infeasible. Without a heuristic, the agent may only receive rewards at the time of failure, or even rewards that guide it away from failures. For example, some approaches give rewards for taking more-likely actions, because we want to find more-likely failures. However, the agent may then learn to only take likely actions, and may not be able to find a failure at all. Consequently, the problem becomes a hard-exploration problem, where rewards do not aid exploration. A new algorithm, go-explore (GE), has recently set new records on benchmarks from the hard-exploration field. We apply GE to adaptive stress testing (AST), one example of an RL-based falsification approach that provides a way to search for the most-likely failure scenario. We simulate a scenario where an autonomous vehicle drives while a pedestrian is crossing the road. We demonstrate that GE is able to find failures without domain-specific heuristics, such as the distance between the car and the pedestrian, on scenarios that other RL techniques are unable to solve. Furthermore, inspired by the robustification phase of GE, we demonstrate that the backwards algorithm (BA) improves the failures found by other RL techniques.

READ FULL TEXT
research
08/02/2019

Adaptive Stress Testing with Reward Augmentation for Autonomous Vehicle Validation

Determining possible failure scenarios is a critical step in the evaluat...
research
07/16/2019

Efficient Autonomy Validation in Simulation with Adaptive Stress Testing

During the development of autonomous systems such as driverless cars, it...
research
08/24/2023

Reinforcement Learning Informed Evolutionary Search for Autonomous Systems Testing

Evolutionary search-based techniques are commonly used for testing auton...
research
07/26/2019

On Hard Exploration for Reinforcement Learning: a Case Study in Pommerman

How to best explore in domains with sparse, delayed, and deceptive rewar...
research
09/30/2021

Reinforcement Learning for Classical Planning: Viewing Heuristics as Dense Reward Generators

Recent advances in reinforcement learning (RL) have led to a growing int...
research
07/08/2021

Adaptive Stress Testing for Adversarial Learning in a Financial Environment

We demonstrate the use of Adaptive Stress Testing to detect and address ...

Please sign up or login with your details

Forgot password? Click here to reset