Bayesian Masking: Sparse Bayesian Estimation with Weaker Shrinkage Bias

09/03/2015
by   Yohei Kondo, et al.
0

A common strategy for sparse linear regression is to introduce regularization, which eliminates irrelevant features by letting the corresponding weights be zeros. However, regularization often shrinks the estimator for relevant features, which leads to incorrect feature selection. Motivated by the above-mentioned issue, we propose Bayesian masking (BM), a sparse estimation method which imposes no regularization on the weights. The key concept of BM is to introduce binary latent variables that randomly mask features. Estimating the masking rates determines the relevance of the features automatically. We derive a variational Bayesian inference algorithm that maximizes the lower bound of the factorized information criterion (FIC), which is a recently developed asymptotic criterion for evaluating the marginal log-likelihood. In addition, we propose reparametrization to accelerate the convergence of the derived algorithm. Finally, we show that BM outperforms Lasso and automatic relevance determination (ARD) in terms of the sparsity-shrinkage trade-off.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/31/2023

Adaptive sparseness for correntropy-based robust regression via automatic relevance determination

Sparseness and robustness are two important properties for many machine ...
research
06/18/2012

Factorized Asymptotic Bayesian Hidden Markov Models

This paper addresses the issue of model selection for hidden Markov mode...
research
05/18/2020

Sparse Methods for Automatic Relevance Determination

This work considers methods for imposing sparsity in Bayesian regression...
research
12/22/2021

Bayesian Approaches to Shrinkage and Sparse Estimation

In all areas of human knowledge, datasets are increasing in both size an...
research
05/06/2018

Bayesian Regularization for Graphical Models with Unequal Shrinkage

We consider a Bayesian framework for estimating a high-dimensional spars...
research
05/24/2016

Relevant sparse codes with variational information bottleneck

In many applications, it is desirable to extract only the relevant aspec...
research
10/29/2020

An Exact Solution Path Algorithm for SLOPE and Quasi-Spherical OSCAR

Sorted L_1 penalization estimator (SLOPE) is a regularization technique ...

Please sign up or login with your details

Forgot password? Click here to reset