Towards a practical lip-to-speech conversion system using deep neural networks and mobile application frontend

04/29/2021
by   Frigyes Viktor Arthur, et al.
0

Articulatory-to-acoustic (forward) mapping is a technique to predict speech using various articulatory acquisition techniques as input (e.g. ultrasound tongue imaging, MRI, lip video). The advantage of lip video is that it is easily available and affordable: most modern smartphones have a front camera. There are already a few solutions for lip-to-speech synthesis, but they mostly concentrate on offline training and inference. In this paper, we propose a system built from a backend for deep neural network training and inference and a fronted as a form of a mobile application. Our initial evaluation shows that the scenario is feasible: a top-5 classification accuracy of 74 with feedback from the mobile application user, making sure that the speaking impaired might be able to communicate with this solution.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 4

page 5

08/06/2020

Ultrasound-based Articulatory-to-Acoustic Mapping with WaveGlow Speech Synthesis

For articulatory-to-acoustic mapping using deep neural networks, typical...
08/03/2020

Speaker dependent articulatory-to-acoustic mapping using real-time MRI of the vocal tract

Articulatory-to-acoustic (forward) mapping is a technique to predict spe...
06/24/2019

Ultrasound-based Silent Speech Interface Built on a Continuous Vocoder

Recently it was shown that within the Silent Speech Interface (SSI) fiel...
07/26/2021

Adaptation of Tacotron2-based Text-To-Speech for Articulatory-to-Acoustic Mapping using Ultrasound Tongue Imaging

For articulatory-to-acoustic mapping, typically only limited parallel tr...
01/10/2019

Handcrafted vs Deep Learning Classification for Scalable Video QoE Modeling

Mobile video traffic is dominant in cellular and enterprise wireless net...
11/24/1998

Speech Synthesis with Neural Networks

Text-to-speech conversion has traditionally been performed either by con...
08/01/2020

BatNet: Data transmission between smartphones over ultrasound

In this paper, we present BatNet, a data transmission mechanism using ul...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.