Learning Representations in Model-Free Hierarchical Reinforcement Learning

10/23/2018
by   Jacob Rafati, et al.
0

Common approaches to Reinforcement Learning (RL) are seriously challenged by large-scale applications involving huge state spaces and sparse delayed reward feedback. Hierarchical Reinforcement Learning (HRL) methods attempt to address this scalability issue by learning action selection policies at multiple levels of temporal abstraction. Abstraction can be had by identifying a relatively small set of states that are likely to be useful as subgoals, in concert with the learning of corresponding skill policies to achieve those subgoals. Many approaches to subgoal discovery in HRL depend on the analysis of a model of the environment, but the need to learn such a model introduces its own problems of scale. Once subgoals are identified, skills may be learned through intrinsic motivation, introducing an internal reward signal marking subgoal attainment. In this paper, we present a novel model-free method for subgoal discovery using incremental unsupervised learning over a small memory of the most recent experiences of the agent. When combined with an intrinsic motivation learning mechanism, this method learns subgoals and skills together, based on experiences in the environment. Thus, we offer an original approach to HRL that does not require the acquisition of a model of the environment, suitable for large-scale applications. We demonstrate the efficiency of our method on two RL problems with sparse delayed feedback: a variant of the rooms environment and the ATARI 2600 game called Montezuma's Revenge.

READ FULL TEXT

page 12

page 15

research
07/02/2019

Dynamics-Aware Unsupervised Discovery of Skills

Conventionally, model-based reinforcement learning (MBRL) aims to learn ...
research
07/14/2020

Efficient Online Estimation of Empowerment for Reinforcement Learning

Training artificial agents to acquire desired skills through model-free ...
research
10/12/2018

Optimal Hierarchical Learning Path Design with Reinforcement Learning

E-learning systems are capable of providing more adaptive and efficient ...
research
05/22/2019

The Journey is the Reward: Unsupervised Learning of Influential Trajectories

Unsupervised exploration and representation learning become increasingly...
research
09/19/2022

An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey

The reinforcement learning (RL) research area is very active, with an im...
research
03/29/2019

Learning Good Representation via Continuous Attention

In this paper we present our scientific discovery that good representati...

Please sign up or login with your details

Forgot password? Click here to reset