Piecewise Latent Variables for Neural Variational Text Processing

12/01/2016
by   Iulian V. Serban, et al.
0

Advances in neural variational inference have facilitated the learning of powerful directed graphical models with continuous latent variables, such as variational autoencoders. The hope is that such models will learn to represent rich, multi-modal latent factors in real-world data, such as natural language text. However, current models often assume simplistic priors on the latent variables - such as the uni-modal Gaussian distribution - which are incapable of representing complex latent factors efficiently. To overcome this restriction, we propose the simple, but highly flexible, piecewise constant distribution. This distribution has the capacity to represent an exponential number of modes of a latent target distribution, while remaining mathematically tractable. Our results demonstrate that incorporating this new latent distribution into different models yields substantial improvements in natural language processing tasks such as document modeling and natural language generation for dialogue.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/25/2021

Conjugate Energy-Based Models

In this paper, we propose conjugate energy-based models (CEBMs), a new c...
research
06/28/2020

Mixture of Discrete Normalizing Flows for Variational Inference

Advances in gradient-based inference have made distributional approximat...
research
05/31/2018

DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encoder

Variational autoencoders (VAEs) have shown a promise in data-driven conv...
research
11/19/2015

Neural Variational Inference for Text Processing

Recent advances in neural variational inference have spawned a renaissan...
research
10/20/2020

Variational Dynamic Mixtures

Deep probabilistic time series forecasting models have become an integra...
research
02/06/2018

Improving Variational Encoder-Decoders in Dialogue Generation

Variational encoder-decoders (VEDs) have shown promising results in dial...
research
05/18/2018

GumBolt: Extending Gumbel trick to Boltzmann priors

Boltzmann machines (BMs) are appealing candidates for powerful priors in...

Please sign up or login with your details

Forgot password? Click here to reset