Wide flat minima and optimal generalization in classifying high-dimensional Gaussian mixtures

10/27/2020
by   Carlo Baldassi, et al.
0

We analyze the connection between minimizers with good generalizing properties and high local entropy regions of a threshold-linear classifier in Gaussian mixtures with the mean squared error loss function. We show that there exist configurations that achieve the Bayes-optimal generalization error, even in the case of unbalanced clusters. We explore analytically the error-counting loss landscape in the vicinity of a Bayes-optimal solution, and show that the closer we get to such configurations, the higher the local entropy, implying that the Bayes-optimal solution lays inside a wide flat region. We also consider the algorithmically relevant case of targeting wide flat minima of the (differentiable) mean squared error loss. Our analytical and numerical results show not only that in the balanced case the dependence on the norm of the weights is mild, but also, in the unbalanced case, that the performances can be improved.

READ FULL TEXT
research
06/14/2020

Entropic gradient descent algorithms and wide flat minima

The properties of flat minima in the empirical risk landscape of neural ...
research
05/20/2019

Shaping the learning landscape in neural networks around wide flat minima

Learning in Deep Neural Networks (DNN) takes place by minimizing a non-c...
research
10/05/2018

Efficient Estimation of Smooth Functionals in Gaussian Shift Models

We study a problem of estimation of smooth functionals of parameter θ o...
research
05/31/2023

Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape

We study the loss landscape of two-layer mildly overparameterized ReLU n...
research
08/22/2023

A free from local minima algorithm for training regressive MLP neural networks

In this article an innovative method for training regressive MLP network...
research
03/15/2021

Evolving parametrized Loss for Image Classification Learning on Small Datasets

This paper proposes a meta-learning approach to evolving a parametrized ...
research
11/14/2019

Empirical Bayes mean estimation with nonparametric errors via order statistic regression

We study empirical Bayes estimation of the effect sizes of N units from ...

Please sign up or login with your details

Forgot password? Click here to reset