Understanding Gesture and Speech Multimodal Interactions for Manipulation Tasks in Augmented Reality Using Unconstrained Elicitation

09/14/2020
by   Adam S. Williams, et al.
0

This research establishes a better understanding of the syntax choices in speech interactions and of how speech, gesture, and multimodal gesture and speech interactions are produced by users in unconstrained object manipulation environments using augmented reality. The work presents a multimodal elicitation study conducted with 24 participants. The canonical referents for translation, rotation, and scale were used along with some abstract referents (create, destroy, and select). In this study time windows for gesture and speech multimodal interactions are developed using the start and stop times of gestures and speech as well as the stoke times for gestures. While gestures commonly precede speech by 81 ms we find that the stroke of the gesture is commonly within 10 ms of the start of speech. Indicating that the information content of a gesture and its co-occurring speech are well aligned to each other. Lastly, the trends across the most common proposals for each modality are examined. Showing that the disagreement between proposals is often caused by a variation of hand posture or syntax. Allowing us to present aliasing recommendations to increase the percentage of users' natural interactions captured by future multimodal interactive systems.

READ FULL TEXT
research
07/25/2022

Eliciting Multimodal Gesture+Speech Interactions in a Multi-Object Augmented Reality Environment

As augmented reality technology and hardware become more mature and affo...
research
08/12/2021

Multimodal analysis of the predictability of hand-gesture properties

Embodied conversational agents benefit from being able to accompany thei...
research
03/16/2020

A Formal Analysis of Multimodal Referring Strategies Under Common Ground

In this paper, we present an analysis of computationally generated mixed...
research
07/11/2012

Automated Training and Maintenance through Kinect

In this paper, we have worked on reducing burden on mechanic involving c...
research
05/20/2023

Efficient Multimodal Neural Networks for Trigger-less Voice Assistants

The adoption of multimodal interactions by Voice Assistants (VAs) is gro...
research
12/03/2020

Multicenter Assessment of Augmented Reality Registration Methods for Image-guided Interventions

Purpose: To evaluate manual and automatic registration times as well as ...
research
09/18/2019

Multimodal Continuation-style Architectures for Human-Robot Interaction

We present an architecture for integrating real-time, multimodal input i...

Please sign up or login with your details

Forgot password? Click here to reset