New Computational and Statistical Aspects of Regularized Regression with Application to Rare Feature Selection and Aggregation

04/10/2019
by   Amin Jalali, et al.
0

Prior knowledge on properties of a target model often come as discrete or combinatorial descriptions. This work provides a unified computational framework for defining norms that promote such structures. More specifically, we develop associated tools for optimization involving such norms given only the orthogonal projection oracle onto the non-convex set of desired models. As an example, we study a norm, which we term the doubly-sparse norm, for promoting vectors with few nonzero entries taking only a few distinct values. We further discuss how the K-means algorithm can serve as the underlying projection oracle in this case and how it can be efficiently represented as a quadratically constrained quadratic program. Our motivation for the study of this norm is regularized regression in the presence of rare features which poses a challenge to various methods within high-dimensional statistics, and in machine learning in general. The proposed estimation procedure is designed to perform automatic feature selection and aggregation for which we develop statistical bounds. The bounds are general and offer a statistical framework for norm-based regularization. The bounds rely on novel geometric quantities on which we attempt to elaborate as well.

READ FULL TEXT
research
02/07/2019

On Mean Estimation for General Norms with Statistical Queries

We study the problem of mean estimation for high-dimensional distributio...
research
09/12/2011

Structured sparsity through convex optimization

Sparse estimation methods are aimed at using or obtaining parsimonious r...
research
05/09/2012

Multi-Task Feature Learning Via Efficient l2,1-Norm Minimization

The problem of joint feature selection across a group of related tasks h...
research
11/08/2017

Clustering with feature selection using alternating minimization, Application to computational biology

This paper deals with unsupervised clustering with feature selection. Th...
research
06/06/2020

Regularized Off-Policy TD-Learning

We present a novel l_1 regularized off-policy convergent TD-learning met...
research
06/18/2022

Tree-Guided Rare Feature Selection and Logic Aggregation with Electronic Health Records Data

Statistical learning with a large number of rare binary features is comm...
research
04/11/2022

Computing a Sparse Projection into a Box

We describe a procedure to compute a projection of w ∈ℝ^n into the inter...

Please sign up or login with your details

Forgot password? Click here to reset