Interactive decoding of words from visual speech recognition models

07/01/2021
by   Brendan Shillingford, et al.
3

This work describes an interactive decoding method to improve the performance of visual speech recognition systems using user input to compensate for the inherent ambiguity of the task. Unlike most phoneme-to-word decoding pipelines, which produce phonemes and feed these through a finite state transducer, our method instead expands words in lockstep, facilitating the insertion of interaction points at each word position. Interaction points enable us to solicit input during decoding, allowing users to interactively direct the decoding process. We simulate the behavior of user input using an oracle to give an automated evaluation, and show promise for the use of this method for text input.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/09/2019

Who Needs Words? Lexicon-Free Speech Recognition

Lexicon-free speech recognition naturally deals with the problem of out-...
research
04/13/2017

Mobile Keyboard Input Decoding with Finite-State Transducers

We propose a finite-state transducer (FST) representation for the models...
research
07/07/2021

Improving Speech Recognition Accuracy of Local POI Using Geographical Models

Nowadays voice search for points of interest (POI) is becoming increasin...
research
12/08/2017

Building competitive direct acoustics-to-word models for English conversational speech recognition

Direct acoustics-to-word (A2W) models in the end-to-end paradigm have re...
research
05/08/2018

Comparing phonemes and visemes with DNN-based lipreading

There is debate if phoneme or viseme units are the most effective for a ...
research
06/08/2020

Learning to Count Words in Fluent Speech enables Online Speech Recognition

Sequence to Sequence models, in particular the Transformer, achieve stat...
research
01/18/2021

Tiny Transducer: A Highly-efficient Speech Recognition Model on Edge Devices

This paper proposes an extremely lightweight phone-based transducer mode...

Please sign up or login with your details

Forgot password? Click here to reset