Incremental Object Grounding Using Scene Graphs

01/06/2022
by   John Seon Keun Yi, et al.
0

Object grounding tasks aim to locate the target object in an image through verbal communications. Understanding human command is an important process needed for effective human-robot communication. However, this is challenging because human commands can be ambiguous and erroneous. This paper aims to disambiguate the human's referring expressions by allowing the agent to ask relevant questions based on semantic data obtained from scene graphs. We test if our agent can use relations between objects from a scene graph to ask semantically relevant questions that can disambiguate the original user command. In this paper, we present Incremental Grounding using Scene Graphs (IGSG), a disambiguation model that uses semantic data from an image scene graph and linguistic structures from a language scene graph to ground objects based on human command. Compared to the baseline, IGSG shows promising results in complex real-world scenes where there are multiple identical target objects. IGSG can effectively disambiguate ambiguous or wrong referring expressions by asking disambiguating questions back to the user.

READ FULL TEXT

page 5

page 7

page 8

research
06/11/2018

Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction

This paper presents INGRESS, a robot system that follows human natural l...
research
05/22/2023

An Abstract Specification of VoxML as an Annotation Language

VoxML is a modeling language used to map natural language expressions in...
research
06/09/2019

Referring Expression Grounding by Marginalizing Scene Graph Likelihood

We focus on the task of grounding referring expressions in images, e.g.,...
research
07/12/2021

Leveraging Explainability for Comprehending Referring Expressions in the Real World

For effective human-robot collaboration, it is crucial for robots to und...
research
05/28/2010

Using Soft Constraints To Learn Semantic Models Of Descriptions Of Shapes

The contribution of this paper is to provide a semantic model (using sof...
research
07/07/2021

LanguageRefer: Spatial-Language Model for 3D Visual Grounding

To realize robots that can understand human instructions and perform mea...
research
08/09/2017

Personalized Cinemagraphs using Semantic Understanding and Collaborative Learning

Cinemagraphs are a compelling way to convey dynamic aspects of a scene. ...

Please sign up or login with your details

Forgot password? Click here to reset