Leveraging Explainability for Comprehending Referring Expressions in the Real World

by   Fethiye Irmak Doğan, et al.

For effective human-robot collaboration, it is crucial for robots to understand requests from users and ask reasonable follow-up questions when there are ambiguities. While comprehending the users' object descriptions in the requests, existing studies have focused on this challenge for limited object categories that can be detected or localized with existing object detection and localization modules. On the other hand, in the wild, it is impossible to limit the object categories that can be encountered during the interaction. To understand described objects and resolve ambiguities in the wild, for the first time, we suggest a method by leveraging explainability. Our method focuses on the active regions of a scene to find the described objects without putting the previous constraints on object categories and natural language instructions. We evaluate our method in varied real-world images and observe that the regions suggested by our method can help resolve ambiguities. When we compare our method with a state-of-the-art baseline, we show that our method performs better in scenes with ambiguous objects which cannot be recognized by existing object detectors.



There are no comments yet.


page 2

page 3

page 6


Learning to Generate Unambiguous Spatial Referring Expressions for Real-World Environments

Referring to objects in a natural and unambiguous manner is crucial for ...

Incremental Object Grounding Using Scene Graphs

Object grounding tasks aim to locate the target object in an image throu...

Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions

Comprehension of spoken natural language is an essential component for r...

Using Depth for Improving Referring Expression Comprehension in Real-World Environments

In a human-robot collaborative task where a robot helps its partner by f...

Using Soft Constraints To Learn Semantic Models Of Descriptions Of Shapes

The contribution of this paper is to provide a semantic model (using sof...

Modelling Observation Correlations for Active Exploration and Robust Object Detection

Today, mobile robots are expected to carry out increasingly complex task...

Objects in Semantic Topology

A more realistic object detection paradigm, Open-World Object Detection,...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.