Return-Based Contrastive Representation Learning for Reinforcement Learning

by   Guoqing Liu, et al.

Recently, various auxiliary tasks have been proposed to accelerate representation learning and improve sample efficiency in deep reinforcement learning (RL). However, existing auxiliary tasks do not take the characteristics of RL problems into consideration and are unsupervised. By leveraging returns, the most important feedback signals in RL, we propose a novel auxiliary task that forces the learnt representations to discriminate state-action pairs with different returns. Our auxiliary loss is theoretically justified to learn representations that capture the structure of a new form of state-action abstraction, under which state-action pairs with similar return distributions are aggregated together. In low data regime, our algorithm outperforms strong baselines on complex tasks in Atari games and DeepMind Control suite, and achieves even better performance when combined with existing auxiliary tasks.



page 1

page 2

page 3

page 4


Learning Temporally-Consistent Representations for Data-Efficient Reinforcement Learning

Deep reinforcement learning (RL) agents that exist in high-dimensional s...

Contrastive Learning as Goal-Conditioned Reinforcement Learning

In reinforcement learning (RL), it is easier to solve a task if given a ...

Learning List-wise Representation in Reinforcement Learning for Ads Allocation with Multiple Auxiliary Tasks

With the recent prevalence of reinforcement learning (RL), there have be...

Learning Representations for Control with Hierarchical Forward Models

Learning control from pixels is difficult for reinforcement learning (RL...

On The Effect of Auxiliary Tasks on Representation Dynamics

While auxiliary tasks play a key role in shaping the representations lea...

Privileged Information Dropout in Reinforcement Learning

Using privileged information during training can improve the sample effi...

Value-Consistent Representation Learning for Data-Efficient Reinforcement Learning

Deep reinforcement learning (RL) algorithms suffer severe performance de...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.