Automatic Risk Adaptation in Distributional Reinforcement Learning

06/11/2021
by   Frederik Schubert, et al.
0

The use of Reinforcement Learning (RL) agents in practical applications requires the consideration of suboptimal outcomes, depending on the familiarity of the agent with its environment. This is especially important in safety-critical environments, where errors can lead to high costs or damage. In distributional RL, the risk-sensitivity can be controlled via different distortion measures of the estimated return distribution. However, these distortion functions require an estimate of the risk level, which is difficult to obtain and depends on the current state. In this work, we demonstrate the suboptimality of a static risk level estimation and propose a method to dynamically select risk levels at each environment step. Our method ARA (Automatic Risk Adaptation) estimates the appropriate risk level in both known and unknown environments using a Random Network Distillation error. We show reduced failure rates by up to a factor of 7 and improved generalization performance by up to 14 in several locomotion environments.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/15/2020

Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings

Reinforcement learning (RL) in real-world safety-critical target setting...
research
11/30/2022

One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning

Offline reinforcement learning (RL) is suitable for safety-critical doma...
research
06/28/2022

Risk Perspective Exploration in Distributional Reinforcement Learning

Distributional reinforcement learning demonstrates state-of-the-art perf...
research
11/12/2021

Two steps to risk sensitivity

Distributional reinforcement learning (RL) – in which agents learn about...
research
08/18/2023

Robust Quadrupedal Locomotion via Risk-Averse Policy Learning

The robustness of legged locomotion is crucial for quadrupedal robots in...
research
03/23/2023

Policy Evaluation in Distributional LQR

Distributional reinforcement learning (DRL) enhances the understanding o...
research
01/13/2023

Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learning

In safety-critical decision-making scenarios being able to identify wors...

Please sign up or login with your details

Forgot password? Click here to reset