Teaching Robots Novel Objects by Pointing at Them

12/25/2020
by   Sagar Gubbi Venkatesh, et al.
3

Robots that must operate in novel environments and collaborate with humans must be capable of acquiring new knowledge from human experts during operation. We propose teaching a robot novel objects it has not encountered before by pointing a hand at the new object of interest. An end-to-end neural network is used to attend to the novel object of interest indicated by the pointing hand and then to localize the object in new scenes. In order to attend to the novel object indicated by the pointing hand, we propose a spatial attention modulation mechanism that learns to focus on the highlighted object while ignoring the other objects in the scene. We show that a robot arm can manipulate novel objects that are highlighted by pointing a hand at them. We also evaluate the performance of the proposed architecture on a synthetic dataset constructed using emojis and on a real-world dataset of common objects.

READ FULL TEXT

page 4

page 5

page 6

research
12/26/2020

One-Shot Object Localization Using Learnt Visual Cues via Siamese Networks

A robot that can operate in novel and unstructured environments must be ...
research
12/26/2020

Spatial Reasoning from Natural Language Instructions for Robot Manipulation

Robots that can manipulate objects in unstructured environments and coll...
research
04/01/2019

Self-Supervised Robot In-hand Object Learning

In order to complete tasks in a new environment, robots must be able to ...
research
08/03/2022

Robots with Different Embodiments Can Express and Influence Carefulness in Object Manipulation

Humans have an extraordinary ability to communicate and read the propert...
research
04/09/2018

Learning at the Ends: From Hand to Tool Affordances in Humanoid Robots

One of the open challenges in designing robots that operate successfully...
research
10/17/2018

Video Segmentation using Teacher-Student Adaptation in a Human Robot Interaction (HRI) Setting

Video segmentation is a challenging task that has many applications in r...
research
05/24/2022

TAILOR: Teaching with Active and Incremental Learning for Object Registration

When deploying a robot to a new task, one often has to train it to detec...

Please sign up or login with your details

Forgot password? Click here to reset