Hallucinating Value: A Pitfall of Dyna-style Planning with Imperfect Environment Models

06/08/2020
by   Taher Jafferjee, et al.
3

Dyna-style reinforcement learning (RL) agents improve sample efficiency over model-free RL agents by updating the value function with simulated experience generated by an environment model. However, it is often difficult to learn accurate models of environment dynamics, and even small errors may result in failure of Dyna agents. In this paper, we investigate one type of model error: hallucinated states. These are states generated by the model, but that are not real states of the environment. We present the Hallucinated Value Hypothesis (HVH): updating values of real states towards values of hallucinated states results in misleading state-action values which adversely affect the control policy. We discuss and evaluate four Dyna variants; three which update real states toward simulated – and therefore potentially hallucinated – states and one which does not. The experimental results provide evidence for the HVH thus suggesting a fruitful direction toward developing Dyna algorithms robust to model error.

READ FULL TEXT
research
10/25/2021

Self-Consistent Models and Values

Learned models of the environment provide reinforcement learning (RL) ag...
research
01/12/2022

Dyna-T: Dyna-Q and Upper Confidence Bounds Applied to Trees

In this work we present a preliminary investigation of a novel algorithm...
research
06/18/2019

Hill Climbing on Value Estimates for Search-control in Dyna

Dyna is an architecture for model-based reinforcement learning (RL), whe...
research
05/30/2020

MM-KTD: Multiple Model Kalman Temporal Differences for Reinforcement Learning

There has been an increasing surge of interest on development of advance...
research
04/26/2023

Can Agents Run Relay Race with Strangers? Generalization of RL to Out-of-Distribution Trajectories

In this paper, we define, evaluate, and improve the “relay-generalizatio...
research
05/22/2022

Should Models Be Accurate?

Model-based Reinforcement Learning (MBRL) holds promise for data-efficie...
research
07/20/2021

Reinforcement learning autonomously identifying the source of errors for agents in a group mission

When agents are swarmed to carry out a mission, there is often a sudden ...

Please sign up or login with your details

Forgot password? Click here to reset