Time Limits in Reinforcement Learning

11/24/2020
by   Fabio Pardo, et al.
0

In reinforcement learning, it is common to let an agent interact for a fixed amount of time with its environment before resetting it and repeating the process in a series of episodes. The task that the agent has to learn can either be to maximize its performance over (i) that fixed period, or (ii) an indefinite period where time limits are only used during training to diversify experience. In this paper, we provide a formal account for how time limits could effectively be handled in each of the two cases and explain why not doing so can cause state-aliasing and invalidation of experience replay, leading to suboptimal policies and training instability. In case (i), we argue that the terminations due to time limits are in fact part of the environment, and thus a notion of the remaining time should be included as part of the agent’s input to avoid violation of the Markov property. In case (ii), the time limits are not part of the environment and are only used to facilitate learning. We argue that this insight should be incorporated by bootstrapping from the value of the state at the end of each partial episode. For both cases, we illustrate empirically the significance of our considerations in improving the performance and stability of existing reinforcement learning algorithms, showing state-of-the-art results on several control tasks.

READ FULL TEXT

page 2

page 4

page 6

page 7

research
05/30/2017

Experience Replay Using Transition Sequences

Experience replay is one of the most commonly used approaches to improve...
research
04/01/1997

Lifeworld Analysis

We argue that the analysis of agent/environment interactions should be e...
research
03/24/2022

Remember and Forget Experience Replay for Multi-Agent Reinforcement Learning

We present the extension of the Remember and Forget for Experience Repla...
research
02/27/2019

Introspection Learning

Traditional reinforcement learning agents learn from experience, past or...
research
11/26/2020

Reinforcement Learning for Robust Missile Autopilot Design

Designing missiles' autopilot controllers has been a complex task, given...
research
10/09/2018

Reinforcement Learning for Improving Agent Design

In many reinforcement learning tasks, the goal is to learn a policy to m...
research
06/16/2021

How memory architecture affects performance and learning in simple POMDPs

Reinforcement learning is made much more complex when the agent's observ...

Please sign up or login with your details

Forgot password? Click here to reset