Tighter Variational Bounds are Not Necessarily Better

02/13/2018
by   Tom Rainforth, et al.
0

We provide theoretical and empirical evidence that using tighter evidence lower bounds (ELBOs) can be detrimental to the process of learning an inference network by reducing the signal-to-noise ratio of the gradient estimator. Our results call into question common implicit assumptions that tighter ELBOs are better variational objectives for simultaneous model learning and inference amortization schemes. Based on our insights, we introduce three new algorithms: the partially importance weighted auto-encoder (PIWAE), the multiply importance weighted auto-encoder (MIWAE), and the combination importance weighted auto-encoder (CIWAE), each of which includes the standard importance weighted auto-encoder (IWAE) as a special case. We show that each can deliver improvements over IWAE, even when performance is measured by the IWAE target itself. Moreover, PIWAE can simultaneously deliver improvements in both the quality of the inference network and generative network, relative to IWAE.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/23/2022

Tighter Variational Bounds are Not Necessarily Better. A Research Report on Implementation, Ablation Study, and Extensions

This report explains, implements and extends the works presented in "Tig...
research
06/07/2019

Importance Weighted Adversarial Variational Autoencoders for Spike Inference from Calcium Imaging Data

The Importance Weighted Auto Encoder (IWAE) objective has been shown to ...
research
06/26/2018

Hierarchical VampPrior Variational Fair Auto-Encoder

Decision making is a process that is extremely prone to different biases...
research
10/12/2022

Alpha-divergence Variational Inference Meets Importance Weighted Auto-Encoders: Methodology and Asymptotics

Several algorithms involving the Variational Rényi (VR) bound have been ...
research
05/24/2022

Gacs-Korner Common Information Variational Autoencoder

We propose a notion of common information that allows one to quantify an...
research
09/15/2023

RVAE-EM: Generative speech dereverberation based on recurrent variational auto-encoder and convolutive transfer function

In indoor scenes, reverberation is a crucial factor in degrading the per...
research
10/16/2020

Extracting Signals of Higgs Boson From Background Noise Using Deep Neural Networks

Higgs boson is a fundamental particle, and the classification of Higgs s...

Please sign up or login with your details

Forgot password? Click here to reset