A Truncated EM Approach for Spike-and-Slab Sparse Coding

11/15/2012
by   Abdul-Saboor Sheikh, et al.
0

We study inference and learning based on a sparse coding model with `spike-and-slab' prior. As in standard sparse coding, the model used assumes independent latent sources that linearly combine to generate data points. However, instead of using a standard sparse prior such as a Laplace distribution, we study the application of a more flexible `spike-and-slab' distribution which models the absence or presence of a source's contribution independently of its strength if it contributes. We investigate two approaches to optimize the parameters of spike-and-slab sparse coding: a novel truncated EM approach and, for comparison, an approach based on standard factored variational distributions. The truncated approach can be regarded as a variational approach with truncated posteriors as variational distributions. In applications to source separation we find that both approaches improve the state-of-the-art in a number of standard benchmarks, which argues for the use of `spike-and-slab' priors for the corresponding data domains. Furthermore, we find that the truncated EM approach improves on the standard factored approach in source separation tasks-which hints to biases introduced by assuming posterior independence in the factored variational approach. Likewise, on a standard benchmark for image denoising, we find that the truncated EM approach improves on the factored variational approach. While the performance of the factored approach saturates with increasing numbers of hidden dimensions, the performance of the truncated approach improves the state-of-the-art for higher noise levels.

READ FULL TEXT
research
05/12/2011

Closed-form EM for Sparse Coding and its Application to Source Separation

We define and discuss the first sparse coding algorithm based on closed-...
research
12/22/2020

Evolutionary Variational Optimization of Generative Models

We combine two popular optimization approaches to derive learning algori...
research
01/16/2012

Spike-and-Slab Sparse Coding for Unsupervised Feature Discovery

We consider the problem of using a factor model we call spike-and-slab ...
research
10/10/2016

Truncated Variational Expectation Maximization

We derive a novel variational expectation maximization approach based on...
research
04/16/2017

k-Means is a Variational EM Approximation of Gaussian Mixture Models

We show that k-means (Lloyd's algorithm) is equivalent to a variational ...
research
06/27/2012

Large-Scale Feature Learning With Spike-and-Slab Sparse Coding

We consider the problem of object recognition with a large number of cla...
research
12/21/2017

Truncated Variational Sampling for "Black Box" Optimization of Generative Models

We investigate the optimization of two generative models with binary hid...

Please sign up or login with your details

Forgot password? Click here to reset