Selective Dyna-style Planning Under Limited Model Capacity

07/05/2020
by   Muhammad Zaheer, et al.
3

In model-based reinforcement learning, planning with an imperfect model of the environment has the potential to harm learning progress. But even when a model is imperfect, it may still contain information that is useful for planning. In this paper, we investigate the idea of using an imperfect model selectively. The agent should plan in parts of the state space where the model would be helpful but refrain from using the model where it would be harmful. An effective selective planning mechanism requires estimating predictive uncertainty, which arises out of aleatoric uncertainty, parameter uncertainty, and model inadequacy, among other sources. Prior work has focused on parameter uncertainty for selective planning. In this work, we emphasize the importance of model inadequacy. We show that heteroscedastic regression can signal predictive uncertainty arising from model inadequacy that is complementary to that which is detected by methods designed for parameter uncertainty, indicating that considering both parameter uncertainty and model inadequacy may be a more promising direction for effective selective planning than either in isolation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/09/2021

Risk Sensitive Model-Based Reinforcement Learning using Uncertainty Guided Planning

Identifying uncertainty and taking mitigating actions is crucial for saf...
research
06/25/2021

Predictive Control Using Learned State Space Models via Rolling Horizon Evolution

A large part of the interest in model-based reinforcement learning deriv...
research
08/25/2017

The Price of Uncertainty in Present-Biased Planning

The tendency to overestimate immediate utility is a common cognitive bia...
research
04/15/2020

Bootstrapped model learning and error correction for planning with uncertainty in model-based RL

Having access to a forward model enables the use of planning algorithms ...
research
06/03/2021

A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learning

We present an end-to-end, model-based deep reinforcement learning agent ...
research
10/11/2017

Uncertainty Averse Pushing with Model Predictive Path Integral Control

Planning robust robot manipulation requires good forward models that ena...
research
02/12/2020

Fast Planning Over Roadmaps via Selective Densification

We propose the Selective Densification method for fast motion planning t...

Please sign up or login with your details

Forgot password? Click here to reset