Improving Speech Inversion Through Self-Supervised Embeddings and Enhanced Tract Variables

09/17/2023

∙

The performance of deep learning models depends significantly on their capacity to encode input features efficiently and decode them into meaningful outputs. Better input and output representation has the potential to boost models' performance and generalization. In the context of acoustic-to-articulatory speech inversion (SI) systems, we study the impact of utilizing speech representations acquired via self-supervised learning (SSL) models, such as HuBERT compared to conventional acoustic features. Additionally, we investigate the incorporation of novel tract variables (TVs) through an improved geometric transformation model. By combining these two approaches, we improve the Pearson product-moment correlation (PPMC) scores which evaluate the accuracy of TV estimation of the SI system from 0.7452 to 0.8141, a 6.9 feature representations from SSL models and improved geometric transformations with target TVs on the enhanced functionality of SI systems.

READ FULL TEXT

Improving Speech Inversion Through Self-Supervised Embeddings and Enhanced Tract Variables

Improved acoustic-to-articulatory inversion using representations from pretrained self-supervised learning models

Supervised Acoustic Embeddings And Their Transferability Across Languages

Acoustic-to-articulatory inversion for dysarthric speech: Are pre-trained self-supervised representations favorable?

Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation model

Evaluation of Speech Representations for MOS prediction

SALTTS: Leveraging Self-Supervised Speech Representations for improved Text-to-Speech Synthesis

Learning to Compute the Articulatory Representations of Speech with the MIRRORNET

Improving Speech Inversion Through Self-Supervised Embeddings and Enhanced Tract Variables

Related Research

Improved acoustic-to-articulatory inversion using representations from pretrained self-supervised learning models

Supervised Acoustic Embeddings And Their Transferability Across Languages

Acoustic-to-articulatory inversion for dysarthric speech: Are pre-trained self-supervised representations favorable?

Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation model

Evaluation of Speech Representations for MOS prediction

SALTTS: Leveraging Self-Supervised Speech Representations for improved Text-to-Speech Synthesis

Learning to Compute the Articulatory Representations of Speech with the MIRRORNET