End-to-End Learning of Semantic Grasping

07/06/2017
by   Eric Jang, et al.
0

We consider the task of semantic robotic grasping, in which a robot picks up an object of a user-specified class using only monocular images. Inspired by the two-stream hypothesis of visual reasoning, we present a semantic grasping framework that learns object detection, classification, and grasp planning in an end-to-end fashion. A "ventral stream" recognizes object class while a "dorsal stream" simultaneously interprets the geometric relationships necessary to execute successful grasps. We leverage the autonomous data collection capabilities of robots to obtain a large self-supervised dataset for training the dorsal stream, and use semi-supervised label propagation to train the ventral stream with only a modest amount of human supervision. We experimentally show that our approach improves upon grasping systems whose components are not learned end-to-end, including a baseline method that uses bounding box detection. Furthermore, we show that jointly training our model with auxiliary data consisting of non-semantic grasping data, as well as semantically labeled images without grasp actions, has the potential to substantially improve semantic grasping performance.

READ FULL TEXT

page 2

page 4

page 5

page 6

page 7

page 11

page 12

page 14

research
11/05/2020

Improving Robotic Grasping on Monocular Images Via Multi-Task Learning and Positional Loss

In this paper, we introduce two methods of improving real-time object gr...
research
11/20/2021

Real-World Semantic Grasping Detection

Reducing the scope of grasping detection according to the semantic infor...
research
06/24/2019

Learning Grasp Affordance Reasoning through Semantic Relations

Reasoning about object affordances allows an autonomous agent to perform...
research
06/30/2019

GarmNet: Improving Global with Local Perception for Robotic Laundry Folding

Developing autonomous assistants to help with domestic tasks is a vital ...
research
05/05/2023

Clothes Grasping and Unfolding Based on RGB-D Semantic Segmentation

Clothes grasping and unfolding is a core step in robotic-assisted dressi...
research
09/28/2016

Learning to Push by Grasping: Using multiple tasks for effective learning

Recently, end-to-end learning frameworks are gaining prevalence in the f...

Please sign up or login with your details

Forgot password? Click here to reset