Sparse Bayesian Learning via Stepwise Regression

06/11/2021
by   Sebastian Ament, et al.
8

Sparse Bayesian Learning (SBL) is a powerful framework for attaining sparsity in probabilistic models. Herein, we propose a coordinate ascent algorithm for SBL termed Relevance Matching Pursuit (RMP) and show that, as its noise variance parameter goes to zero, RMP exhibits a surprising connection to Stepwise Regression. Further, we derive novel guarantees for Stepwise Regression algorithms, which also shed light on RMP. Our guarantees for Forward Regression improve on deterministic and probabilistic results for Orthogonal Matching Pursuit with noise. Our analysis of Backward Regression on determined systems culminates in a bound on the residual of the optimal solution to the subset selection problem that, if satisfied, guarantees the optimality of the result. To our knowledge, this bound is the first that can be computed in polynomial time and depends chiefly on the smallest singular value of the matrix. We report numerical experiments using a variety of feature selection algorithms. Notably, RMP and its limiting variant are both efficient and maintain strong performance with correlated features.

READ FULL TEXT
research
06/06/2021

On the Optimality of Backward Regression: Sparse Recovery and Subset Selection

Sparse recovery and subset selection are fundamental problems in varied ...
research
06/02/2018

Signal and Noise Statistics Oblivious Orthogonal Matching Pursuit

Orthogonal matching pursuit (OMP) is a widely used algorithm for recover...
research
11/22/2020

Online Orthogonal Matching Pursuit

Greedy algorithms for feature selection are widely used for recovering s...
research
05/01/2019

Restricted Connection Orthogonal Matching Pursuit For Sparse Subspace Clustering

Sparse Subspace Clustering (SSC) is one of the most popular methods for ...
research
07/27/2017

Signal and Noise Statistics Oblivious Sparse Reconstruction using OMP/OLS

Orthogonal matching pursuit (OMP) and orthogonal least squares (OLS) are...
research
08/22/2020

A Modified Orthogonal Matching Pursuit for Construction of Sparse Probabilistic Boolean Networks

The probabilistic Boolean network plays a remarkable role in the modelli...

Please sign up or login with your details

Forgot password? Click here to reset