End-to-End Learning of Semantic Grasping

by   Eric Jang, et al.
The Team at X

We consider the task of semantic robotic grasping, in which a robot picks up an object of a user-specified class using only monocular images. Inspired by the two-stream hypothesis of visual reasoning, we present a semantic grasping framework that learns object detection, classification, and grasp planning in an end-to-end fashion. A "ventral stream" recognizes object class while a "dorsal stream" simultaneously interprets the geometric relationships necessary to execute successful grasps. We leverage the autonomous data collection capabilities of robots to obtain a large self-supervised dataset for training the dorsal stream, and use semi-supervised label propagation to train the ventral stream with only a modest amount of human supervision. We experimentally show that our approach improves upon grasping systems whose components are not learned end-to-end, including a baseline method that uses bounding box detection. Furthermore, we show that jointly training our model with auxiliary data consisting of non-semantic grasping data, as well as semantically labeled images without grasp actions, has the potential to substantially improve semantic grasping performance.


page 2

page 4

page 5

page 6

page 7

page 11

page 12

page 14


Improving Robotic Grasping on Monocular Images Via Multi-Task Learning and Positional Loss

In this paper, we introduce two methods of improving real-time object gr...

Real-World Semantic Grasping Detection

Reducing the scope of grasping detection according to the semantic infor...

Learning Grasp Affordance Reasoning through Semantic Relations

Reasoning about object affordances allows an autonomous agent to perform...

GarmNet: Improving Global with Local Perception for Robotic Laundry Folding

Developing autonomous assistants to help with domestic tasks is a vital ...

Clothes Grasping and Unfolding Based on RGB-D Semantic Segmentation

Clothes grasping and unfolding is a core step in robotic-assisted dressi...

Learning to Push by Grasping: Using multiple tasks for effective learning

Recently, end-to-end learning frameworks are gaining prevalence in the f...

Please sign up or login with your details

Forgot password? Click here to reset