Dual-Space Analysis of the Sparse Linear Model

07/10/2012
by   David Wipf, et al.
0

Sparse linear (or generalized linear) models combine a standard likelihood function with a sparse prior on the unknown coefficients. These priors can conveniently be expressed as a maximization over zero-mean Gaussians with different variance hyperparameters. Standard MAP estimation (Type I) involves maximizing over both the hyperparameters and coefficients, while an empirical Bayesian alternative (Type II) first marginalizes the coefficients and then maximizes over the hyperparameters, leading to a tractable posterior approximation. The underlying cost functions can be related via a dual-space framework from Wipf et al. (2011), which allows both the Type I or Type II objectives to be expressed in either coefficient or hyperparmeter space. This perspective is useful because some analyses or extensions are more conducive to development in one space or the other. Herein we consider the estimation of a trade-off parameter balancing sparsity and data fit. As this parameter is effectively a variance, natural estimators exist by assessing the problem in hyperparameter (variance) space, transitioning natural ideas from Type II to solve what is much less intuitive for Type I. In contrast, for analyses of update rules and sparsity properties of local and global solutions, as well as extensions to more general likelihood models, we can leverage coefficient-space techniques developed for Type I and apply them to Type II. For example, this allows us to prove that Type II-inspired techniques can be successful recovering sparse coefficients when unfavorable restricted isometry properties (RIP) lead to failure of popular L1 reconstructions. It also facilitates the analysis of Type II when non-Gaussian likelihood models lead to intractable integrations.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/04/2022

Sparse Gaussian Process Hyperparameters: Optimize or Integrate?

The kernel function and its hyperparameters are the central model select...
research
08/22/2011

Sparse Estimation using Bayesian Hierarchical Prior Modeling for Real and Complex Linear Models

In sparse Bayesian learning (SBL), Gaussian scale mixtures (GSMs) have b...
research
08/15/2022

Intuitive Joint Priors for Bayesian Linear Multilevel Models: The R2D2M2 prior

The training of high-dimensional regression models on comparably sparse ...
research
05/25/2023

Bayesian Analysis for Over-parameterized Linear Model without Sparsity

In high-dimensional Bayesian statistics, several methods have been devel...
research
05/26/2022

A proof of consistency and model-selection optimality on the empirical Bayes method

We study the consistency and optimality of the maximum marginal likeliho...
research
10/29/2021

Support Recovery with Stochastic Gates: Theory and Application for Linear Models

We analyze the problem of simultaneous support recovery and estimation o...

Please sign up or login with your details

Forgot password? Click here to reset