Strategic Attentive Writer for Learning Macro-Actions

06/15/2016
by   Alexander, et al.
0

We present a novel deep recurrent neural network architecture that learns to build implicit plans in an end-to-end manner by purely interacting with an environment in reinforcement learning setting. The network builds an internal plan, which is continuously updated upon observation of the next input from the environment. It can also partition this internal representation into contiguous sub- sequences by learning for how long the plan can be committed to - i.e. followed without re-planing. Combining these properties, the proposed model, dubbed STRategic Attentive Writer (STRAW) can learn high-level, temporally abstracted macro- actions of varying lengths that are solely learnt from data without any prior information. These macro-actions enable both structured exploration and economic computation. We experimentally demonstrate that STRAW delivers strong improvements on several ATARI games by employing temporally extended planning strategies (e.g. Ms. Pacman and Frostbite). It is at the same time a general algorithm that can be applied on any sequence data. To that end, we also show that when trained on text prediction task, STRAW naturally predicts frequent n-grams (instead of macro-actions), demonstrating the generality of the approach.

READ FULL TEXT

page 3

page 6

page 8

research
10/07/2016

Learning Macro-actions for State-Space Planning

Planning has achieved significant progress in recent years. Among the va...
research
10/22/2018

Mining useful Macro-actions in Planning

Planning has achieved significant progress in recent years. Among the va...
research
03/22/2019

Macro Action Reinforcement Learning with Sequence Disentanglement using Variational Autoencoder

One problem in the application of reinforcement learning to real-world p...
research
08/16/2023

Planning in the imagination: High-level planning on learned abstract search spaces

We propose a new method, called PiZero, that gives an agent the ability ...
research
11/07/2020

MAGIC: Learning Macro-Actions for Online POMDP Planning using Generator-Critic

When robots operate in the real-world, they need to handle uncertainties...
research
11/28/2017

Plan, Attend, Generate: Planning for Sequence-to-Sequence Models

We investigate the integration of a planning mechanism into sequence-to-...
research
12/13/2019

Long-Term Planning and Situational Awareness in OpenAI Five

Understanding how knowledge about the world is represented within model-...

Please sign up or login with your details

Forgot password? Click here to reset