Real-World Robot Learning with Masked Visual Pre-training

10/06/2022
by   Ilija Radosavovic, et al.
15

In this work, we explore self-supervised visual pre-training on images from diverse, in-the-wild videos for real-world robotic tasks. Like prior work, our visual representations are pre-trained via a masked autoencoder (MAE), frozen, and then passed into a learnable control module. Unlike prior work, we show that the pre-trained representations are effective across a range of real-world robotic tasks and embodiments. We find that our encoder consistently outperforms CLIP (up to 75 training from scratch (up to 81 transformer on a massive collection of 4.5M images from the Internet and egocentric videos, and demonstrate clearly the benefits of scaling visual pre-training for robot learning.

READ FULL TEXT

page 2

page 3

page 4

page 6

page 8

page 9

page 10

research
03/11/2022

Masked Visual Pre-training for Motor Control

This paper shows that self-supervised visual pre-training from real-worl...
research
08/07/2023

Exploring Visual Pre-training for Robot Manipulation: Datasets, Models and Methods

Visual pre-training with large-scale real-world data has made great prog...
research
06/16/2023

Robot Learning with Sensorimotor Pre-training

We present a self-supervised sensorimotor pre-training approach for robo...
research
07/07/2023

SpawnNet: Learning Generalizable Visuomotor Skills from Pre-trained Networks

The existing internet-scale image and video datasets cover a wide range ...
research
09/22/2022

PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Training

Robotics has long been a field riddled with complex systems architecture...
research
03/31/2023

Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?

We present the largest and most comprehensive empirical study of pre-tra...
research
08/04/2022

Impact Makes a Sound and Sound Makes an Impact: Sound Guides Representations and Explorations

Sound is one of the most informative and abundant modalities in the real...

Please sign up or login with your details

Forgot password? Click here to reset