On mechanisms for transfer using landmark value functions in multi-task lifelong reinforcement learning

07/01/2019
by   Nick Denis, et al.
0

Transfer learning across different reinforcement learning (RL) tasks is becoming an increasingly valuable area of research. We consider a goal-based multi-task RL framework and mechanisms by which previously solved tasks can reduce sample complexity and regret when the agent is faced with a new task. Specifically, we introduce two metrics on the state space that encode notions of traversibility of the state space for an agent. Using these metrics a topological covering is constructed by way of a set of landmark states in a fully self-supervised manner. We show that these landmark coverings confer theoretical advantages for transfer learning within the goal-based multi-task RL setting. Specifically, we demonstrate three mechanisms by which landmark coverings can be used for successful transfer learning. First, we extend the Landmark Options Via Reflection (LOVR) framework to this new topological covering; second, we use the landmark-centric value functions themselves as features and define a greedy zombie policy that achieves near oracle performance on a sequence of zero-shot transfer tasks; finally, motivated by the second transfer mechanism, we introduce a learned reward function that provides a more dense reward signal for goal-based RL. Our novel topological landmark covering confers beneficial theoretical results, bounding the Q values at each state-action pair. In doing so, we introduce a mechanism that performs action-pruning at infeasible actions which cannot possibly be part of an optimal policy for the current goal.

READ FULL TEXT

page 8

page 9

research
08/17/2020

On the Sample Complexity of Reinforcement Learning with Policy Space Generalization

We study the optimal sample complexity in large-scale Reinforcement Lear...
research
02/25/2020

Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement

Multi-task reinforcement learning (RL) aims to simultaneously learn poli...
research
11/28/2022

Inapplicable Actions Learning for Knowledge Transfer in Reinforcement Learning

Reinforcement Learning (RL) algorithms are known to scale poorly to envi...
research
07/15/2020

Model-Based Multi-Agent RL in Zero-Sum Markov Games with Near-Optimal Sample Complexity

Model-based reinforcement learning (RL), which finds an optimal policy u...
research
02/10/2023

Robust Knowledge Transfer in Tiered Reinforcement Learning

In this paper, we study the Tiered Reinforcement Learning setting, a par...
research
02/26/2020

Generalized Hindsight for Reinforcement Learning

One of the key reasons for the high sample complexity in reinforcement l...
research
03/07/2016

Learning Shared Representations in Multi-task Reinforcement Learning

We investigate a paradigm in multi-task reinforcement learning (MT-RL) i...

Please sign up or login with your details

Forgot password? Click here to reset