Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation

12/09/2021
by   Anthony Simeonov, et al.
7

We present Neural Descriptor Fields (NDFs), an object representation that encodes both points and relative poses between an object and a target (such as a robot gripper or a rack used for hanging) via category-level descriptors. We employ this representation for object manipulation, where given a task demonstration, we want to repeat the same task on a new object instance from the same category. We propose to achieve this objective by searching (via optimization) for the pose whose descriptor matches that observed in the demonstration. NDFs are conveniently trained in a self-supervised fashion via a 3D auto-encoding task that does not rely on expert-labeled keypoints. Further, NDFs are SE(3)-equivariant, guaranteeing performance that generalizes across all possible 3D object translations and rotations. We demonstrate learning of manipulation tasks from few (5-10) demonstrations both in simulation and on a real robot. Our performance generalizes across both object instances and 6-DoF object poses, and significantly outperforms a recent baseline that relies on 2D descriptors. Project website: https://yilundu.github.io/ndf/.

READ FULL TEXT

page 1

page 5

page 8

research
02/07/2023

Local Neural Descriptor Fields: Locally Conditioned Object Representations for Manipulation

A robot operating in a household environment will see a wide range of un...
research
11/17/2022

SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields

We present a method for performing tasks involving spatial relations bet...
research
06/16/2022

Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learning

End-to-end learning for visual robotic manipulation is known to suffer f...
research
03/17/2021

Learning Descriptor of Constrained Task from Demonstration

Constrained objects, such as doors and drawers are often complex and sha...
research
08/29/2023

Robot Manipulation Task Learning by Leveraging SE(3) Group Invariance and Equivariance

This paper presents a differential geometric control approach that lever...
research
01/30/2022

You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration

Promising results have been achieved recently in category-level manipula...
research
08/01/2022

Robust Change Detection Based on Neural Descriptor Fields

The ability to reason about changes in the environment is crucial for ro...

Please sign up or login with your details

Forgot password? Click here to reset