
The role of regularization in classification of highdimensional noisy Gaussian mixture
We consider a highdimensional mixture of two Gaussians in the noisy reg...
read it

Dimension Independent Generalization Error with Regularized Online Optimization
One classical canon of statistics is that large models are prone to over...
read it

An Empirical Bayes Approach for High Dimensional Classification
We propose an empirical Bayes estimator based on Dirichlet process mixtu...
read it

Fundamental Barriers to HighDimensional Regression with Convex Penalties
In highdimensional regression, we attempt to estimate a parameter vecto...
read it

Large scale analysis of generalization error in learning using margin based classification methods
Largemargin classifiers are popular methods for classification. We deri...
read it

Kernel Truncated Randomized Ridge Regression: Optimal Rates and Low Noise Acceleration
In this paper, we consider the nonparametric least square regression in ...
read it

Wide flat minima and optimal generalization in classifying highdimensional Gaussian mixtures
We analyze the connection between minimizers with good generalizing prop...
read it
Generalization error in highdimensional perceptrons: Approaching Bayes error with convex optimization
We consider a commonly studied supervised classification of a synthetic dataset whose labels are generated by feeding a onelayer neural network with random iid inputs. We study the generalization performances of standard classifiers in the highdimensional regime where α=n/d is kept finite in the limit of a high dimension d and number of samples n. Our contribution is threefold: First, we prove a formula for the generalization error achieved by ℓ_2 regularized classifiers that minimize a convex loss. This formula was first obtained by the heuristic replica method of statistical physics. Secondly, focussing on commonly used loss functions and optimizing the ℓ_2 regularization strength, we observe that while ridge regression performance is poor, logistic and hinge regression are surprisingly able to approach the Bayesoptimal generalization error extremely closely. As α→∞ they lead to Bayesoptimal rates, a fact that does not follow from predictions of marginbased generalization error bounds. Third, we design an optimal loss and regularizer that provably leads to Bayesoptimal generalization error.
READ FULL TEXT
Comments
There are no comments yet.