Prosody Based Co-analysis for Continuous Recognition of Coverbal Gestures

11/05/2002
by   Sanshzar Kettebekov, et al.
0

Although speech and gesture recognition has been studied extensively, all the successful attempts of combining them in the unified framework were semantically motivated, e.g., keyword-gesture cooccurrence. Such formulations inherited the complexity of natural language processing. This paper presents a Bayesian formulation that uses a phenomenon of gesture and speech articulation for improving accuracy of automatic recognition of continuous coverbal gestures. The prosodic features from the speech signal were coanalyzed with the visual signal to learn the prior probability of co-occurrence of the prominent spoken segments with the particular kinematical phases of gestures. It was found that the above co-analysis helps in detecting and disambiguating visually small gestures, which subsequently improves the rate of continuous gesture recognition. The efficacy of the proposed approach was demonstrated on a large database collected from the weather channel broadcast. This formulation opens new avenues for bottom-up frameworks of multimodal integration.

READ FULL TEXT
research
05/17/2001

Toward Natural Gesture/Speech Control of a Large Display

In recent years because of the advances in computer vision research, fre...
research
03/23/2023

GesGPT: Speech Gesture Synthesis With Text Parsing from GPT

Gesture synthesis has gained significant attention as a critical researc...
research
03/04/2021

It's A Match! Gesture Generation Using Expressive Parameter Matching

Automatic gesture generation from speech generally relies on implicit mo...
research
12/02/2015

Continuous and Simultaneous Gesture and Posture Recognition for Commanding a Robotic Wheelchair; Towards Spotting the Signal Patterns

Spotting signal patterns with varying lengths has been still an open pro...
research
08/17/2020

Sequence-to-Sequence Predictive Model: From Prosody To Communicative Gestures

Communicative gestures and speech prosody are tightly linked. Our object...
research
10/25/2017

High Five: Improving Gesture Recognition by Embracing Uncertainty

Sensors on mobile devices---accelerometers, gyroscopes, pressure meters,...
research
08/21/2023

Co-Speech Gesture Detection through Multi-phase Sequence Labeling

Gestures are integral components of face-to-face communication. They unf...

Please sign up or login with your details

Forgot password? Click here to reset