Variational Gibbs inference for statistical model estimation from incomplete data

by   Vaidotas Simkus, et al.

Statistical models are central to machine learning with broad applicability across a range of downstream tasks. The models are typically controlled by free parameters that are estimated from data by maximum-likelihood estimation. However, when faced with real-world datasets many of the models run into a critical issue: they are formulated in terms of fully-observed data, whereas in practice the datasets are plagued with missing data. The theory of statistical model estimation from incomplete data is conceptually similar to the estimation of latent-variable models, where powerful tools such as variational inference (VI) exist. However, in contrast to standard latent-variable models, parameter estimation with incomplete data often requires estimating exponentially-many conditional distributions of the missing variables, hence making standard VI methods intractable. We address this gap by introducing variational Gibbs inference (VGI), a new general-purpose method to estimate the parameters of statistical models from incomplete data. We validate VGI on a set of synthetic and real-world estimation tasks, estimating important machine learning models, VAEs and normalising flows, from incomplete data. The proposed method, whilst general-purpose, achieves competitive or better performance than existing model-specific estimation methods.



There are no comments yet.


page 2

page 3

page 27


Variational Noise-Contrastive Estimation

Unnormalised latent variable models are a broad and flexible class of st...

Online but Accurate Inference for Latent Variable Models with Local Gibbs Sampling

We study parameter inference in large-scale latent variable models. We f...

Reconstruction of Incomplete Wildfire Data using Deep Generative Models

We present our submission to the Extreme Value Analysis 2021 Data Challe...

A General Method for Amortizing Variational Filtering

We introduce the variational filtering EM algorithm, a simple, general-p...

A Machine Learning Approach for Flagging Incomplete Bid-rigging Cartels

We propose a new method for flagging bid rigging, which is particularly ...

A New Approach to Building the Interindustry Input--Output Table

We present a new approach to estimating the interdependence of industrie...

Solving Non-identifiable Latent Feature Models

Latent feature models (LFM)s are widely employed for extracting latent s...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.