Environment Predictive Coding for Embodied Agents

by   Santhosh K. Ramakrishnan, et al.

We introduce environment predictive coding, a self-supervised approach to learn environment-level representations for embodied agents. In contrast to prior work on self-supervised learning for images, we aim to jointly encode a series of images gathered by an agent as it moves about in 3D environments. We learn these representations via a zone prediction task, where we intelligently mask out portions of an agent's trajectory and predict them from the unmasked portions, conditioned on the agent's camera poses. By learning such representations on a collection of videos, we demonstrate successful transfer to multiple downstream navigation-oriented tasks. Our experiments on the photorealistic 3D environments of Gibson and Matterport3D show that our method outperforms the state-of-the-art on challenging tasks with only a limited budget of experience.


page 2

page 4

page 5

page 6


Supervise Thyself: Examining Self-Supervised Representations in Interactive Environments

Self-supervised methods, wherein an agent learns representations solely ...

Self-Supervised Domain Adaptation for Visual Navigation with Global Map Consistency

We propose a light-weight, self-supervised adaptation for a visual navig...

Data-Efficient Reinforcement Learning with Momentum Predictive Representations

While deep reinforcement learning excels at solving tasks where large am...

Explore and Explain: Self-supervised Navigation and Recounting

Embodied AI has been recently gaining attention as it aims to foster the...

Character Generation through Self-Supervised Vectorization

The prevalent approach in self-supervised image generation is to operate...

A Self-Supervised Framework for Function Learning and Extrapolation

Understanding how agents learn to generalize – and, in particular, to ex...

Self-supervised Body Image Acquisition Using a Deep Neural Network for Sensorimotor Prediction

This work investigates how a naive agent can acquire its own body image ...