Log In Sign Up

Variational Gibbs inference for statistical model estimation from incomplete data

by   Vaidotas Simkus, et al.

Statistical models are central to machine learning with broad applicability across a range of downstream tasks. The models are typically controlled by free parameters that are estimated from data by maximum-likelihood estimation. However, when faced with real-world datasets many of the models run into a critical issue: they are formulated in terms of fully-observed data, whereas in practice the datasets are plagued with missing data. The theory of statistical model estimation from incomplete data is conceptually similar to the estimation of latent-variable models, where powerful tools such as variational inference (VI) exist. However, in contrast to standard latent-variable models, parameter estimation with incomplete data often requires estimating exponentially-many conditional distributions of the missing variables, hence making standard VI methods intractable. We address this gap by introducing variational Gibbs inference (VGI), a new general-purpose method to estimate the parameters of statistical models from incomplete data. We validate VGI on a set of synthetic and real-world estimation tasks, estimating important machine learning models, VAEs and normalising flows, from incomplete data. The proposed method, whilst general-purpose, achieves competitive or better performance than existing model-specific estimation methods.


page 2

page 3

page 27


Variational Noise-Contrastive Estimation

Unnormalised latent variable models are a broad and flexible class of st...

Online but Accurate Inference for Latent Variable Models with Local Gibbs Sampling

We study parameter inference in large-scale latent variable models. We f...

Reconstruction of Incomplete Wildfire Data using Deep Generative Models

We present our submission to the Extreme Value Analysis 2021 Data Challe...

A General Method for Amortizing Variational Filtering

We introduce the variational filtering EM algorithm, a simple, general-p...

A Machine Learning Approach for Flagging Incomplete Bid-rigging Cartels

We propose a new method for flagging bid rigging, which is particularly ...

A New Approach to Building the Interindustry Input--Output Table

We present a new approach to estimating the interdependence of industrie...