DeepAI AI Chat
Log In Sign Up

Multiresolution and Multimodal Speech Recognition with Transformers

by   Georgios Paraskevopoulos, et al.
National Technical University of Athens

This paper presents an audio visual automatic speech recognition (AV-ASR) system using a Transformer-based architecture. We particularly focus on the scene context provided by the visual information, to ground the ASR. We extract representations for audio features in the encoder layers of the transformer and fuse video features using an additional crossmodal multihead attention layer. Additionally, we incorporate a multitask training criterion for multiresolution ASR, where we train the model to generate both character and subword level transcriptions. Experimental results on the How2 dataset, indicate that multiresolution training can speed up convergence by around 50 error rate (WER) performance by upto 18 Further, incorporating visual information improves performance with relative gains upto 3.76 Our results are comparable to state-of-the-art Listen, Attend and Spell-based architectures.


page 1

page 2

page 3

page 4


Audio-Visual Speech Recognition is Worth 32×32×8 Voxels

Audio-visual automatic speech recognition (AV-ASR) introduces the video ...

MM-ALT: A Multimodal Automatic Lyric Transcription System

Automatic lyric transcription (ALT) is a nascent field of study attracti...

Transformers with convolutional context for ASR

The recent success of transformer networks for neural machine translatio...

Voice Quality and Pitch Features in Transformer-Based Speech Recognition

Jitter and shimmer measurements have shown to be carriers of voice quali...

WHALETRANS: E2E WHisper to nAturaL spEech conversion using modified TRANSformer network

In this article, we investigate whispered-to natural-speech conversion m...

TVLT: Textless Vision-Language Transformer

In this work, we present the Textless Vision-Language Transformer (TVLT)...