VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors

10/20/2022
by   Yifeng Zhu, et al.
0

We introduce VIOLA, an object-centric imitation learning approach to learning closed-loop visuomotor policies for robot manipulation. Our approach constructs object-centric representations based on general object proposals from a pre-trained vision model. VIOLA uses a transformer-based policy to reason over these representations and attend to the task-relevant visual factors for action prediction. Such object-based structural priors improve deep imitation learning algorithm's robustness against object variations and environmental perturbations. We quantitatively evaluate VIOLA in simulation and on real robots. VIOLA outperforms the state-of-the-art imitation learning methods by 45.8% in success rate. It has also been deployed successfully on a physical robot to solve challenging long-horizon tasks, such as dining table arrangement and coffee making. More videos and model details can be found in supplementary material and the project website: https://ut-austin-rpl.github.io/VIOLA .

READ FULL TEXT

page 6

page 8

research
02/10/2022

Memory-based gaze prediction in deep imitation learning for robot manipulation

Deep imitation learning is a promising approach that does not require ha...
research
05/25/2023

Imitating Task and Motion Planning with Visuomotor Transformers

Imitation learning is a powerful tool for training robot manipulation po...
research
11/06/2020

RetinaGAN: An Object-aware Approach to Sim-to-Real Transfer

The success of deep reinforcement learning (RL) and imitation learning (...
research
07/26/2023

Waypoint-Based Imitation Learning for Robotic Manipulation

While imitation learning methods have seen a resurgent interest for robo...
research
05/15/2019

Simitate: A Hybrid Imitation Learning Benchmark

We present Simitate --- a hybrid benchmarking suite targeting the evalua...
research
02/21/2020

The Surprising Effectiveness of Linear Models for Visual Foresight in Object Pile Manipulation

In this paper, we tackle the problem of pushing piles of small objects i...
research
06/14/2023

Unraveling the ARC Puzzle: Mimicking Human Solutions with Object-Centric Decision Transformer

In the pursuit of artificial general intelligence (AGI), we tackle Abstr...

Please sign up or login with your details

Forgot password? Click here to reset