DeepAI AI Chat
Log In Sign Up

Interactive Robotic Grasping with Attribute-Guided Disambiguation

03/15/2022
by   Yang Yang, et al.
University of Minnesota
0

Interactive robotic grasping using natural language is one of the most fundamental tasks in human-robot interaction. However, language can be a source of ambiguity, particularly when there are ambiguous visual or linguistic contents. This paper investigates the use of object attributes in disambiguation and develops an interactive grasping system capable of effectively resolving ambiguities via dialogues. Our approach first predicts target scores and attribute scores through vision-and-language grounding. To handle ambiguous objects and commands, we propose an attribute-guided formulation of the partially observable Markov decision process (Attr-POMDP) for disambiguation. The Attr-POMDP utilizes target and attribute scores as the observation model to calculate the expected return of an attribute-based (e.g., "what is the color of the target, red or green?") or a pointing-based (e.g., "do you mean this one?") question. Our disambiguation module runs in real time on a real robot, and the interactive grasping system achieves a 91.43% selection accuracy in the real-robot experiments, outperforming several baselines by large margins.

READ FULL TEXT

page 1

page 3

page 5

page 6

08/25/2021

INVIGORATE: Interactive Visual Grounding and Grasping in Clutter

This paper presents INVIGORATE, a robot system that interacts with human...
06/06/2021

Planning Multimodal Exploratory Actions for Online Robot Attribute Learning

Robots frequently need to perceive object attributes, such as "red," "he...
05/28/2018

Interactive Text2Pickup Network for Natural Language based Human-Robot Collaboration

In this paper, we propose the Interactive Text2Pickup (IT2P) network for...
01/27/2023

Learning 6-DoF Fine-grained Grasp Detection Based on Part Affordance Grounding

Robotic grasping is a fundamental ability for a robot to interact with t...
05/09/2022

Learning 6-DoF Object Poses to Grasp Category-level Objects by Language Instructions

This paper studies the task of any objects grasping from the known categ...
01/24/2023

Generalized Object Search

Future collaborative robots must be capable of finding objects. As such ...
05/03/2017

Concurrent Constraint Conditional-Branching Timed Interactive Scores

Multimedia scenarios have multimedia content and interactive events asso...