Risk Sensitive Model-Based Reinforcement Learning using Uncertainty Guided Planning

11/09/2021
by   Stefan Radic Webster, et al.
24

Identifying uncertainty and taking mitigating actions is crucial for safe and trustworthy reinforcement learning agents, especially when deployed in high-risk environments. In this paper, risk sensitivity is promoted in a model-based reinforcement learning algorithm by exploiting the ability of a bootstrap ensemble of dynamics models to estimate environment epistemic uncertainty. We propose uncertainty guided cross-entropy method planning, which penalises action sequences that result in high variance state predictions during model rollouts, guiding the agent to known areas of the state space with low uncertainty. Experiments display the ability for the agent to identify uncertain regions of the state space during planning and to take actions that maintain the agent within high confidence areas, without the requirement of explicit constraints. The result is a reduction in the performance in terms of attaining reward, displaying a trade-off between risk and return.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/23/2019

Estimating Risk and Uncertainty in Deep Reinforcement Learning

This paper demonstrates a novel method for separately estimating aleator...
research
07/05/2020

Selective Dyna-style Planning Under Limited Model Capacity

In model-based reinforcement learning, planning with an imperfect model ...
research
10/17/2022

On Uncertainty in Deep State Space Models for Model-Based Reinforcement Learning

Improved state space models, such as Recurrent State Space Models (RSSMs...
research
04/25/2018

Generative Temporal Models with Spatial Memory for Partially Observed Environments

In model-based reinforcement learning, generative and temporal models of...
research
04/15/2020

Bootstrapped model learning and error correction for planning with uncertainty in model-based RL

Having access to a forward model enables the use of planning algorithms ...
research
09/15/2019

Model Based Planning with Energy Based Models

Model-based planning holds great promise for improving both sample effic...
research
04/19/2020

Model-Predictive Control via Cross-Entropy and Gradient-Based Optimization

Recent works in high-dimensional model-predictive control and model-base...

Please sign up or login with your details

Forgot password? Click here to reset