Benefits of Overparameterization in Single-Layer Latent Variable Generative Models

by   Rares-Darius Buhai, et al.

One of the most surprising and exciting discoveries in supervising learning was the benefit of overparametrization (i.e. training a very large model) to improving the optimization landscape of a problem, with minimal effect on statistical performance (i.e. generalization). In contrast, unsupervised settings have been under-explored, despite the fact that it has been observed that overparameterization can be helpful as early as Dasgupta & Schulman (2007). In this paper, we perform an exhaustive study of different aspects of overparameterization in unsupervised learning via synthetic and semi-synthetic experiments. We discuss benefits to different metrics of success (held-out log-likelihood, recovering the parameters of the ground-truth model), sensitivity to variations of the training algorithm, and behavior as the amount of overparameterization increases. We find that, when learning using methods such as variational inference, larger models can significantly increase the number of ground truth latent variables recovered.


page 8

page 19


Accuracy of Latent-Variable Estimation in Bayesian Semi-Supervised Learning

Hierarchical probabilistic models, such as Gaussian mixture models, are ...

Causal Structure Discovery between Clusters of Nodes Induced by Latent Factors

We consider the problem of learning the structure of a causal directed a...

Density estimation using Real NVP

Unsupervised learning of probabilistic models is a central yet challengi...

An Information-Theoretic Analysis of Deep Latent-Variable Models

We present an information-theoretic framework for understanding trade-of...

Fair Inference for Discrete Latent Variable Models

It is now well understood that machine learning models, trained on data ...

Learning multi-modal generative models with permutation-invariant encoders and tighter variational bounds

Devising deep latent variable models for multi-modal data has been a lon...

An Unsupervised Bayesian Neural Network for Truth Discovery

The problem of estimating event truths from conflicting agent opinions i...

Please sign up or login with your details

Forgot password? Click here to reset