Learning Abstract Models for Strategic Exploration and Fast Reward Transfer

07/12/2020
by   Evan Zheran Liu, et al.
12

Model-based reinforcement learning (RL) is appealing because (i) it enables planning and thus more strategic exploration, and (ii) by decoupling dynamics from rewards, it enables fast transfer to new reward functions. However, learning an accurate Markov Decision Process (MDP) over high-dimensional states (e.g., raw pixels) is extremely challenging because it requires function approximation, which leads to compounding errors. Instead, to avoid compounding errors, we propose learning an abstract MDP over abstract states: low-dimensional coarse representations of the state (e.g., capturing agent position, ignoring other objects). We assume access to an abstraction function that maps the concrete states to abstract states. In our approach, we construct an abstract MDP, which grows through strategic exploration via planning. Similar to hierarchical RL approaches, the abstract actions of the abstract MDP are backed by learned subpolicies that navigate between abstract states. Our approach achieves strong results on three of the hardest Arcade Learning Environment games (Montezuma's Revenge, Pitfall!, and Private Eye), including superhuman performance on Pitfall! without demonstrations. After training on one task, we can reuse the learned abstract MDP for new reward functions, achieving higher reward in 1000x fewer samples than model-free methods trained from scratch.

READ FULL TEXT

page 2

page 14

page 15

12/16/2019

Planning with Abstract Learned Models While Learning Transferable Subtasks

We introduce an algorithm for model-based hierarchical reinforcement lea...
05/11/2020

TOMA: Topological Map Abstraction for Reinforcement Learning

Animals are able to discover the topological map (graph) of surrounding ...
12/19/2014

Grounding Hierarchical Reinforcement Learning Models for Knowledge Transfer

Methods of deep machine learning enable to to reuse low-level representa...
03/04/2022

Online Learning of Reusable Abstract Models for Object Goal Navigation

In this paper, we present a novel approach to incrementally learn an Abs...
03/15/2012

Variance-Based Rewards for Approximate Bayesian Reinforcement Learning

The exploreexploit dilemma is one of the central challenges in Reinforce...
11/22/2021

Component Transfer Learning for Deep RL Based on Abstract Representations

In this work we investigate a specific transfer learning approach for de...
10/21/2017

Insulin Regimen ML-based control for T2DM patients

We model individual T2DM patient blood glucose level (BGL) by stochasti...