Degrees of freedom for off-the-grid sparse estimation

11/08/2019
by   Clarice Poon, et al.
6

A central question in modern machine learning and imaging sciences is to quantify the number of effective parameters of vastly over-parameterized models. The degrees of freedom is a mathematically convenient way to define this number of parameters. Its computation and properties are well understood when dealing with discretized linear models, possibly regularized using sparsity. In this paper, we argue that this way of thinking is plagued when dealing with models having very large parameter spaces. In this case it makes more sense to consider "off-the-grid" approaches, using a continuous parameter space. This type of approach is the one favoured when training multi-layer perceptrons, and is also becoming popular to solve super-resolution problems in imaging. Training these off-the-grid models with a sparsity inducing prior can be achieved by solving a convex optimization problem over the space of measures, which is often called the Beurling Lasso (Blasso), and is the continuous counterpart of the celebrated Lasso parameter selection method. In previous works, the degrees of freedom for the Lasso was shown to coincide with the size of the smallest solution support. Our main contribution is a proof of a continuous counterpart to this result for the Blasso. Our findings suggest that discretized methods actually vastly over-estimate the number of intrinsic continuous degrees of freedom. Our second contribution is a detailed study of the case of sampling Fourier coefficients in 1D, which corresponds to a super-resolution problem. We show that our formula for the degrees of freedom is valid outside of a set of measure zero of observations, which in turn justifies its use to compute an unbiased estimator of the prediction risk using the Stein Unbiased Risk Estimator (SURE).

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/30/2016

Degrees of Freedom in Deep Neural Networks

In this paper, we explore degrees of freedom in deep sigmoidal neural ne...
research
05/22/2012

A lasso for hierarchical interactions

We add a set of convex constraints to the lasso to produce sparse intera...
research
10/13/2014

Convex Modeling of Interactions with Strong Heredity

We consider the task of fitting a regression model involving interaction...
research
02/23/2018

A Dual Certificates Analysis of Compressive Off-the-Grid Recovery

Many problems in machine learning and imaging can be framed as an infini...
research
09/18/2014

Fused Lasso Additive Model

We consider the problem of predicting an outcome variable using p covari...
research
03/01/2018

Diversity and degrees of freedom in regression ensembles

Ensemble methods are a cornerstone of modern machine learning. The perfo...
research
06/13/2022

Sparse-group boosting – Unbiased group and variable selection

In the presence of grouped covariates, we propose a framework for boosti...

Please sign up or login with your details

Forgot password? Click here to reset