DeepAI AI Chat
Log In Sign Up

MDP Abstraction with Successor Features

by   Dongge Han, et al.

Abstraction plays an important role for generalisation of knowledge and skills, and is key to sample efficient learning and planning. For many complex problems an abstract plan can be formed first, which is then instantiated by filling in the necessary low-level details. Often, such abstract plans generalize well to related new problems. We study abstraction in the context of reinforcement learning, in which agents may perform state or temporal abstractions. Temporal abstractions aka options represent temporally-extended actions in the form of option policies. However, typically acquired option policies cannot be directly transferred to new environments due to changes in the state space or transition dynamics. Furthermore, many existing state abstraction schemes ignore the correlation between state and temporal abstraction. In this work, we propose successor abstraction, a novel abstraction scheme building on successor features. This includes an algorithm for encoding and instantiation of abstract options across different environments, and a state abstraction mechanism based on the abstract options. Our successor abstraction allows us to learn abstract environment models with semantics that are transferable across different environments through encoding and instantiation of abstract options. Empirically, we achieve better transfer and improved performance on a set of benchmark tasks as compared to relevant state of the art baselines.


page 6

page 19

page 22

page 23

page 24


The Option-Critic Architecture

Temporal abstraction is key to scaling up learning and planning in reinf...

A Direct Approximation of AIXI Using Logical State Abstractions

We propose a practical integration of logical state abstraction with AIX...

Computing and Proving Well-founded Orderings through Finite Abstractions

A common technique for checking properties of complex state machines is ...

Temporally Abstract Partial Models

Humans and animals have the ability to reason and make predictions about...

Causal Dynamics Learning for Task-Independent State Abstraction

Learning dynamics models accurately is an important goal for Model-Based...

Data-efficient Hindsight Off-policy Option Learning

Solutions to most complex tasks can be decomposed into simpler, intermed...

Learning Abstract and Transferable Representations for Planning

We are concerned with the question of how an agent can acquire its own r...