A Real2Sim2Real Method for Robust Object Grasping with Neural Surface Reconstruction

10/06/2022
by   Luobin Wang, et al.
1

Recent 3D-based manipulation methods either directly predict the grasp pose using 3D neural networks, or solve the grasp pose using similar objects retrieved from shape databases. However, the former faces generalizability challenges when testing with new robot arms or unseen objects; and the latter assumes that similar objects exist in the databases. We hypothesize that recent 3D modeling methods provides a path towards building digital replica of the evaluation scene that affords physical simulation and supports robust manipulation algorithm learning. We propose to reconstruct high-quality meshes from real-world point clouds using state-of-the-art neural surface reconstruction method (the Real2Sim step). Because most simulators take meshes for fast simulation, the reconstructed meshes enable grasp pose labels generation without human efforts. The generated labels can train grasp network that performs robustly in the real evaluation scene (the Sim2Real step). In synthetic and real experiments, we show that the Real2Sim2Real pipeline performs better than baseline grasp networks trained with a large dataset and a grasp sampling method with retrieval-based reconstruction. The benefit of the Real2Sim2Real pipeline comes from 1) decoupling scene modeling and grasp sampling into sub-problems, and 2) both sub-problems can be solved with sufficiently high quality using recent 3D learning algorithms and mesh-based physical simulation techniques.

READ FULL TEXT

page 1

page 2

page 4

page 6

research
08/07/2016

Deep Learning a Grasp Function for Grasping under Gripper Pose Uncertainty

This paper presents a new method for parallel-jaw grasping of isolated o...
research
03/21/2020

PointNet++ Grasping: Learning An End-to-end Spatial Grasp Generation Algorithm from Sparse Point Clouds

Grasping for novel objects is important for robot manipulation in unstru...
research
05/25/2019

6-DOF GraspNet: Variational Grasp Generation for Object Manipulation

Generating grasp poses is a crucial component for any robot object manip...
research
10/11/2022

Digital twins for city simulation: Automatic, efficient, and robust mesh generation for large-scale city modeling and simulation

The concept of creating digital twins, connected digital models of physi...
research
01/18/2023

OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation

Recent advances in modeling 3D objects mostly rely on synthetic datasets...
research
02/21/2023

Constrained Generative Sampling of 6-DoF Grasps

Most state-of-the-art data-driven grasp sampling methods propose stable ...
research
08/15/2023

Grasp Transfer based on Self-Aligning Implicit Representations of Local Surfaces

Objects we interact with and manipulate often share similar parts, such ...

Please sign up or login with your details

Forgot password? Click here to reset