Exploring Temporal Dependencies in Multimodal Referring Expressions with Mixed Reality

02/04/2019
by   Elena Sibirtseva, et al.
0

In collaborative tasks, people rely both on verbal and non-verbal cues simultaneously to communicate with each other. For human-robot interaction to run smoothly and naturally, a robot should be equipped with the ability to robustly disambiguate referring expressions. In this work, we propose a model that can disambiguate multimodal fetching requests using modalities such as head movements, hand gestures, and speech. We analysed the acquired data from mixed reality experiments and formulated a hypothesis that modelling temporal dependencies of events in these three modalities increases the model's predictive power. We evaluated our model on a Bayesian framework to interpret referring expressions with and without exploiting a temporal prior.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 2

page 8

page 9

08/11/2017

Communicating Robot Arm Motion Intent Through Mixed Reality Head-mounted Displays

Efficient motion intent communication is necessary for safe and collabor...
07/04/2019

Multimodal Uncertainty Reduction for Intention Recognition in Human-Robot Interaction

Assistive robots can potentially improve the quality of life and persona...
01/14/2021

Enabling Robots to Draw and Tell: Towards Visually Grounded Multimodal Description Generation

Socially competent robots should be equipped with the ability to perceiv...
10/13/2020

Jointly Optimizing Sensing Pipelines for Multimodal Mixed Reality Interaction

Natural human interactions for Mixed Reality Applications are overwhelmi...
01/26/2018

A Comparison of Visualisation Methods for Disambiguating Verbal Requests in Human-Robot Interaction

Picking up objects requested by a human user is a common task in human-r...
11/21/2019

Using Socially Expressive Mixed Reality Arms for Enhancing Low-Expressivity Robots

Expressivity–the use of multiple modalities to convey internal state and...
09/18/2019

Multimodal Continuation-style Architectures for Human-Robot Interaction

We present an architecture for integrating real-time, multimodal input i...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.