Transfer with Model Features in Reinforcement Learning

by   Lucas Lehnert, et al.

A key question in Reinforcement Learning is which representation an agent can learn to efficiently reuse knowledge between different tasks. Recently the Successor Representation was shown to have empirical benefits for transferring knowledge between tasks with shared transition dynamics. This paper presents Model Features: a feature representation that clusters behaviourally equivalent states and that is equivalent to a Model-Reduction. Further, we present a Successor Feature model which shows that learning Successor Features is equivalent to learning a Model-Reduction. A novel optimization objective is developed and we provide bounds showing that minimizing this objective results in an increasingly improved approximation of a Model-Reduction. Further, we provide transfer experiments on randomly generated MDPs which vary in their transition and reward functions but approximately preserve behavioural equivalence between states. These results demonstrate that Model Features are suitable for transfer between tasks with varying transition and reward functions.


page 1

page 2

page 3

page 4


Advantages and Limitations of using Successor Features for Transfer in Reinforcement Learning

One question central to Reinforcement Learning is how to learn a feature...

Successor Features Support Model-based and Model-free Reinforcement Learning

One key challenge in reinforcement learning is the ability to generalize...

Successor Features for Transfer in Reinforcement Learning

Transfer in reinforcement learning refers to the notion that generalizat...

Xi-Learning: Successor Feature Transfer Learning for General Reward Functions

Transfer in Reinforcement Learning aims to improve learning performance ...

A New Representation of Successor Features for Transfer across Dissimilar Environments

Transfer in reinforcement learning is usually achieved through generalis...

Does Knowledge Transfer Always Help to Learn a Better Policy?

One of the key approaches to save samples when learning a policy for a r...

Successor Feature Neural Episodic Control

A longstanding goal in reinforcement learning is to build intelligent ag...