Sim-To-Real Transfer of Visual Grounding for Human-Aided Ambiguity Resolution

05/24/2022
by   Georgios Tziafas, et al.
0

Service robots should be able to interact naturally with non-expert human users, not only to help them in various tasks but also to receive guidance in order to resolve ambiguities that might be present in the instruction. We consider the task of visual grounding, where the agent segments an object from a crowded scene given a natural language description. Modern holistic approaches to visual grounding usually ignore language structure and struggle to cover generic domains, therefore relying heavily on large datasets. Additionally, their transfer performance in RGB-D datasets suffers due to high visual discrepancy between the benchmark and the target domains. Modular approaches marry learning with domain modeling and exploit the compositional nature of language to decouple visual representation from language parsing, but either rely on external parsers or are trained in an end-to-end fashion due to the lack of strong supervision. In this work, we seek to tackle these limitations by introducing a fully decoupled modular framework for compositional visual grounding of entities, attributes, and spatial relations. We exploit rich scene graph annotations generated in a synthetic domain and train each module independently. Our approach is evaluated both in simulation and in two real RGB-D scene datasets. Experimental results show that the decoupled nature of our framework allows for easy integration with domain adaptation approaches for Sim-To-Real visual recognition, offering a data-efficient, robust, and interpretable solution to visual grounding in robotic applications.

READ FULL TEXT

page 1

page 2

page 6

page 9

page 14

research
06/05/2019

Learning to Compose and Reason with Language Tree Structures for Visual Grounding

Grounding natural language in images, such as localizing "the black dog ...
research
09/05/2021

Modular Framework for Visuomotor Language Grounding

Natural language instruction following tasks serve as a valuable test-be...
research
10/01/2022

Differentiable Parsing and Visual Grounding of Verbal Instructions for Object Placement

Grounding spatial relations in natural language for object placing could...
research
11/30/2016

Modeling Relationships in Referential Expressions with Compositional Modular Networks

People often refer to entities in an image in terms of their relationshi...
research
03/17/2021

Few-Shot Visual Grounding for Natural Human-Robot Interaction

Natural Human-Robot Interaction (HRI) is one of the key components for s...
research
10/03/2022

A Hybrid Compositional Reasoning Approach for Interactive Robot Manipulation

In this paper we present a neuro-symbolic (hybrid) compositional reasoni...
research
12/08/2018

Explainability by Parsing: Neural Module Tree Networks for Natural Language Visual Grounding

Grounding natural language in images essentially requires composite visu...

Please sign up or login with your details

Forgot password? Click here to reset