SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo

06/30/2021
by   Thomas Kollar, et al.
0

Robot manipulation of unknown objects in unstructured environments is a challenging problem due to the variety of shapes, materials, arrangements and lighting conditions. Even with large-scale real-world data collection, robust perception and manipulation of transparent and reflective objects across various lighting conditions remain challenging. To address these challenges we propose an approach to performing sim-to-real transfer of robotic perception. The underlying model, SimNet, is trained as a single multi-headed neural network using simulated stereo data as input and simulated object segmentation masks, 3D oriented bounding boxes (OBBs), object keypoints, and disparity as output. A key component of SimNet is the incorporation of a learned stereo sub-network that predicts disparity. SimNet is evaluated on 2D car detection, unknown object detection, and deformable object keypoint detection and significantly outperforms a baseline that uses a structured light RGB-D sensor. By inferring grasp positions using the OBB and keypoint predictions, SimNet can be used to perform end-to-end manipulation of unknown objects in both easy and hard scenarios using our fleet of Toyota HSR robots in four home environments. In unknown object grasping experiments, the predictions from the baseline RGB-D network and SimNet enable successful grasps of most of the easy objects. However, the RGB-D baseline only grasps 35 objects, while SimNet grasps 95 manipulation of unknown objects, including transparent objects, in unknown environments.

READ FULL TEXT

page 6

page 13

page 14

page 15

page 16

page 17

page 18

page 19

research
02/22/2023

MVTrans: Multi-View Perception of Transparent Objects

Transparent object perception is a crucial skill for applications such a...
research
09/10/2019

GlassLoc: Plenoptic Grasp Pose Detection in Transparent Clutter

Transparent objects are prevalent across many environments of interest f...
research
06/07/2022

Learning Mobile Manipulation

Providing mobile robots with the ability to manipulate objects has, desp...
research
03/09/2021

Deep 6-DoF Tracking of Unknown Objects for Reactive Grasping

Robotic manipulation of unknown objects is an important field of researc...
research
10/06/2019

ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation

Transparent objects are a common part of everyday life, yet they possess...
research
07/11/2023

TRansPose: Large-Scale Multispectral Dataset for Transparent Object

Transparent objects are encountered frequently in our daily lives, yet r...
research
05/07/2022

Category-Independent Articulated Object Tracking with Factor Graphs

Robots deployed in human-centric environments may need to manipulate a d...

Please sign up or login with your details

Forgot password? Click here to reset