Learning the joint distribution of two sequences using little or no paired data

12/06/2022
by   Soroosh Mariooryad, et al.
0

We present a noisy channel generative model of two sequences, for example text and speech, which enables uncovering the association between the two modalities when limited paired data is available. To address the intractability of the exact model under a realistic data setup, we propose a variational inference approximation. To train this variational model with categorical data, we propose a KL encoder loss approach which has connections to the wake-sleep algorithm. Identifying the joint or conditional distributions by only observing unpaired samples from the marginals is only possible under certain conditions in the data distribution and we discuss under what type of conditional independence assumptions that might be achieved, which guides the architecture designs. Experimental results show that even tiny amount of paired data (5 minutes) is sufficient to learn to relate the two modalities (graphemes and phonemes here) when a massive amount of unpaired data is available, paving the path to adopting this principled approach for all seq2seq models in low data resource regimes.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/29/2018

Forward Amortized Inference for Likelihood-Free Variational Marginalization

In this paper, we introduce a new form of amortized variational inferenc...
research
10/27/2022

Virtuoso: Massive Multilingual Speech-Text Joint Semi-Supervised Learning for Text-To-Speech

This paper proposes Virtuoso, a massively multilingual speech-text joint...
research
04/21/2022

Learn from Unpaired Data for Image Restoration: A Variational Bayes Approach

Collecting paired training data is difficult in practice, but the unpair...
research
04/13/2019

End-to-end Text-to-speech for Low-resource Languages by Cross-Lingual Transfer Learning

End-to-end text-to-speech (TTS) has shown great success on large quantit...
research
09/08/2020

Learning more expressive joint distributions in multimodal variational methods

Data often are formed of multiple modalities, which jointly describe the...
research
06/04/2019

KERMIT: Generative Insertion-Based Modeling for Sequences

We present KERMIT, a simple insertion-based approach to generative model...

Please sign up or login with your details

Forgot password? Click here to reset