DeepAI AI Chat
Log In Sign Up

Universal Successor Features Approximators

by   Diana Borsa, et al.

The ability of a reinforcement learning (RL) agent to learn about many reward functions at the same time has many potential benefits, such as the decomposition of complex tasks into simpler ones, the exchange of information between tasks, and the reuse of skills. We focus on one aspect in particular, namely the ability to generalise to unseen tasks. Parametric generalisation relies on the interpolation power of a function approximator that is given the task description as input; one of its most common form are universal value function approximators (UVFAs). Another way to generalise to new tasks is to exploit structure in the RL problem itself. Generalised policy improvement (GPI) combines solutions of previous tasks into a policy for the unseen task; this relies on instantaneous policy evaluation of old policies under the new reward function, which is made possible through successor features (SFs). Our proposed universal successor features approximators (USFAs) combine the advantages of all of these, namely the scalability of UVFAs, the instant inference of SFs, and the strong generalisation of GPI. We discuss the challenges involved in training a USFA, its generalisation properties and demonstrate its practical benefits and transfer abilities on a large-scale domain in which the agent has to navigate in a first-person perspective three-dimensional environment.


Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement

The ability to transfer skills across tasks has the potential to scale u...

Will it Blend? Composing Value Functions in Reinforcement Learning

An important property for lifelong-learning agents is the ability to com...

Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer

In many real-world applications, reinforcement learning (RL) agents migh...

Universal Successor Representations for Transfer Reinforcement Learning

The objective of transfer reinforcement learning is to generalize from a...

Universal Successor Features for Transfer Reinforcement Learning

Transfer in Reinforcement Learning (RL) refers to the idea of applying k...

Successor Feature Neural Episodic Control

A longstanding goal in reinforcement learning is to build intelligent ag...

Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration

Curiosity-based reward schemes can present powerful exploration mechanis...

Code Repositories


Implementation of USFAs:

view repo