Voice Reconstruction from Silent Speech with a Sequence-to-Sequence Model

07/31/2021

∙

Silent Speech Decoding (SSD) based on Surface electromyography (sEMG) has become a prevalent task in recent years. Though revolutions have been proposed to decode sEMG to audio successfully, some problems still remain. In this paper, we propose an optimized sequence-to-sequence (Seq2Seq) approach to synthesize voice from subvocal sEMG. Both subvocal and vocal sEMG are collected and preprocessed to provide data information. Then, we extract durations from the alignment between subvocal and vocal signals to regulate the subvocal sEMG following audio length. Besides, we use phoneme classification and vocal sEMG reconstruction modules to improve the model performance. Finally, experiments on a Mandarin speaker dataset, which consists of 6.49 hours of data, demonstrate that the proposed model improves the mapping accuracy and voice quality of reconstructed voice.

READ FULL TEXT

Voice Reconstruction from Silent Speech with a Sequence-to-Sequence Model

Direct speech-to-speech translation with a sequence-to-sequence model

Hierarchical Sequence to Sequence Voice Conversion with Limited Data

Data Efficient Voice Cloning from Noisy Samples with Domain Adversarial Training

Any-to-One Sequence-to-Sequence Voice Conversion using Self-Supervised Discrete Speech Representations

Adversarially Trained Multi-Singer Sequence-To-Sequence Singing Synthesizer

Improving the Performance of Online Neural Transducer Models

Leveraging Symmetrical Convolutional Transformer Networks for Speech to Singing Voice Style Transfer

Voice Reconstruction from Silent Speech with a Sequence-to-Sequence Model

Related Research

Direct speech-to-speech translation with a sequence-to-sequence model

Hierarchical Sequence to Sequence Voice Conversion with Limited Data

Data Efficient Voice Cloning from Noisy Samples with Domain Adversarial Training

Any-to-One Sequence-to-Sequence Voice Conversion using Self-Supervised Discrete Speech Representations

Adversarially Trained Multi-Singer Sequence-To-Sequence Singing Synthesizer

Improving the Performance of Online Neural Transducer Models

Leveraging Symmetrical Convolutional Transformer Networks for Speech to Singing Voice Style Transfer