A New Re-synchronization Method based Multi-modal Fusion for Automatic Continuous Cued Speech Recognition

01/03/2020
by   Li Liu, et al.
0

Cued Speech (CS) is an augmented lip reading complemented by hand coding, and it is very helpful to the deaf people. Automatic CS recognition can help communications between the deaf people and others. Due to the asynchronous nature of lips and hand movements, fusion of them in automatic CS recognition is a challenging problem. In this work, we propose a novel re-synchronization procedure for multi-modal fusion, which aligns the hand features with lips feature. It is realized by delaying hand position and hand shape with their optimal hand preceding time which is derived by investigating the temporal organizations of hand position and hand shape movements in CS. This re-synchronization procedure is incorporated into a practical continuous CS recognition system that combines convolutional neural network (CNN) with multi-stream hidden markov model (MSHMM). A significant improvement of about 4.6% has been achieved retaining 76.6% CS phoneme recognition correctness compared with the state-of-the-art architecture (72.04%), which did not take into account the asynchrony of multi-modal fusion in CS. To our knowledge, this is the first work to tackle the asynchronous multi-modal fusion in the automatic continuous CS recognition.

READ FULL TEXT

page 1

page 4

page 9

page 13

research
01/03/2020

Re-synchronization using the Hand Preceding Model for Multi-modal Fusion in Automatic Continuous Cued Speech Recognition

Cued Speech (CS) is an augmented lip reading complemented by hand coding...
research
06/05/2023

A Novel Interpretable and Generalizable Re-synchronization Model for Cued Speech based on a Multi-Cuer Corpus

Cued Speech (CS) is a multi-modal visual coding system combining lip rea...
research
12/02/2022

Cross-Modal Mutual Learning for Cued Speech Recognition

Automatic Cued Speech Recognition (ACSR) provides an intelligent human-m...
research
06/26/2021

An Attention Self-supervised Contrastive Learning based Three-stage Model for Hand Shape Feature Representation in Cued Speech

Cued Speech (CS) is a communication system for deaf people or hearing im...
research
06/14/2023

Investigating the dynamics of hand and lips in French Cued Speech using attention mechanisms and CTC-based decoding

Hard of hearing or profoundly deaf people make use of cued speech (CS) a...
research
01/03/2020

A Pilot Study on Mandarin Chinese Cued Speech

Cued Speech (CS) is a communication system developed for deaf people, wh...

Please sign up or login with your details

Forgot password? Click here to reset