Trust the Model When It Is Confident: Masked Model-based Actor-Critic

10/10/2020
by   Feiyang Pan, et al.
11

It is a popular belief that model-based Reinforcement Learning (RL) is more sample efficient than model-free RL, but in practice, it is not always true due to overweighed model errors. In complex and noisy settings, model-based RL tends to have trouble using the model if it does not know when to trust the model. In this work, we find that better model usage can make a huge difference. We show theoretically that if the use of model-generated data is restricted to state-action pairs where the model error is small, the performance gap between model and real rollouts can be reduced. It motivates us to use model rollouts only when the model is confident about its predictions. We propose Masked Model-based Actor-Critic (M2AC), a novel policy optimization algorithm that maximizes a model-based lower-bound of the true value function. M2AC implements a masking mechanism based on the model's uncertainty to decide whether its prediction should be used or not. Consequently, the new algorithm tends to give robust policy improvements. Experiments on continuous control benchmarks demonstrate that M2AC has strong performance even when using long model rollouts in very noisy environments, and it significantly outperforms previous state-of-the-art methods.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/29/2020

How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimization

Deterministic-policy actor-critic algorithms for continuous control impr...
research
07/25/2022

Live in the Moment: Learning Dynamics Model Adapted to Evolving Policy

Model-based reinforcement learning (RL) achieves higher sample efficienc...
research
05/16/2020

Model-Augmented Actor-Critic: Backpropagating through Paths

Current model-based reinforcement learning approaches use the model simp...
research
12/16/2021

Sample-Efficient Reinforcement Learning via Conservative Model-Based Actor-Critic

Model-based reinforcement learning algorithms, which aim to learn a mode...
research
05/30/2022

Stock Trading Optimization through Model-based Reinforcement Learning with Resistance Support Relative Strength

Reinforcement learning (RL) is gaining attention by more and more resear...
research
07/06/2023

A Neuromorphic Architecture for Reinforcement Learning from Real-Valued Observations

Reinforcement Learning (RL) provides a powerful framework for decision-m...
research
04/28/2020

Sample-Efficient Model-based Actor-Critic for an Interactive Dialogue Task

Human-computer interactive systems that rely on machine learning are bec...

Please sign up or login with your details

Forgot password? Click here to reset