Probabilistic Best Subset Selection by Gradient-Based Optimization

06/11/2020
by   Mingzhang Yin, et al.
0

In high-dimensional statistics, variable selection is an optimization problem aiming to recover the latent sparse pattern from all possible covariate combinations. In this paper, we transform the optimization problem from a discrete space to a continuous one via reparameterization. The new objective function is a reformulation of the exact L_0-regularized regression problem (a.k.a. best subset selection). In the framework of stochastic gradient descent, we propose a family of unbiased and efficient gradient estimators that are used to optimize the best subset selection objective and its variational lower bound. Under this family, we identify the estimator with non-vanishing signal-to-noise ratio and uniformly minimum variance. Theoretically we study the general conditions under which the method is guaranteed to converge to the ground truth in expectation. In a wide variety of synthetic and real data sets, the proposed method outperforms existing ones based on penalized regression or best subset selection, in both sparse pattern recovery and out-of-sample prediction. Our method can find the true regression model from thousands of covariates in a couple of seconds.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/11/2020

Probabilistic Best Subset Selection via Gradient-Based Optimization

In high-dimensional statistics, variable selection is an optimization pr...
research
12/05/2011

On best subset regression

In this paper we discuss the variable selection method from ℓ0-norm cons...
research
05/05/2022

COMBSS: Best Subset Selection via Continuous Optimization

We consider the problem of best subset selection in linear regression, w...
research
07/14/2020

A Unified Computational and Theoretical Framework for High-Dimensional Bayesian Additive Models

We introduce a general framework for estimation and variable selection i...
research
12/31/2022

On High dimensional Poisson models with measurement error: hypothesis testing for nonlinear nonconvex optimization

We study estimation and testing in the Poisson regression model with noi...
research
08/10/2017

Subset Selection with Shrinkage: Sparse Linear Modeling when the SNR is low

We study the behavior of a fundamental tool in sparse statistical modeli...
research
10/04/2022

SIMPLE: A Gradient Estimator for k-Subset Sampling

k-subset sampling is ubiquitous in machine learning, enabling regulariza...

Please sign up or login with your details

Forgot password? Click here to reset