Learning a Single Neuron with Adversarial Label Noise via Gradient Descent

06/17/2022
โˆ™
by   Ilias Diakonikolas, et al.
โˆ™
0
โˆ™

We study the fundamental problem of learning a single neuron, i.e., a function of the form ๐ฑโ†ฆฯƒ(๐ฐยท๐ฑ) for monotone activations ฯƒ:โ„โ†ฆโ„, with respect to the L_2^2-loss in the presence of adversarial label noise. Specifically, we are given labeled examples from a distribution D on (๐ฑ, y)โˆˆโ„^d ร—โ„ such that there exists ๐ฐ^โˆ—โˆˆโ„^d achieving F(๐ฐ^โˆ—)=ฯต, where F(๐ฐ)=๐„_(๐ฑ,y)โˆผ D[(ฯƒ(๐ฐยท๐ฑ)-y)^2]. The goal of the learner is to output a hypothesis vector ๐ฐ such that F(๐•จ)=C ฯต with high probability, where C>1 is a universal constant. As our main contribution, we give efficient constant-factor approximate learners for a broad class of distributions (including log-concave distributions) and activation functions. Concretely, for the class of isotropic log-concave distributions, we obtain the following important corollaries: For the logistic activation, we obtain the first polynomial-time constant factor approximation (even under the Gaussian distribution). Our algorithm has sample complexity O(d/ฯต), which is tight within polylogarithmic factors. For the ReLU activation, we give an efficient algorithm with sample complexity ร•(d (1/ฯต)). Prior to our work, the best known constant-factor approximate learner had sample complexity ฮฉฬƒ(d/ฯต). In both of these settings, our algorithms are simple, performing gradient-descent on the (regularized) L_2^2-loss. The correctness of our algorithms relies on novel structural results that we establish, showing that (essentially all) stationary points of the underlying non-convex loss are approximately optimal.

READ FULL TEXT
โˆ™ 12/19/2020

On the Power of Localized Perceptron for Label-Optimal Learning of Halfspaces with Adversarial Noise

We study online active learning of homogeneous halfspaces in โ„^d with ad...
โˆ™ 05/29/2020

Agnostic Learning of a Single Neuron with Gradient Descent

We consider the problem of learning the best-fitting single neuron as me...
โˆ™ 05/26/2020

Approximation Schemes for ReLU Regression

We consider the fundamental problem of ReLU regression, where the goal i...
โˆ™ 02/13/2020

Learning Halfspaces with Massart Noise Under Structured Distributions

We study the problem of learning halfspaces with Massart noise in the di...
โˆ™ 08/17/2021

Statistically Near-Optimal Hypothesis Selection

Hypothesis Selection is a fundamental distribution learning problem wher...
โˆ™ 11/06/2012

Active and passive learning of linear separators under log-concave distributions

We provide new results concerning label efficient, polynomial time, pass...
โˆ™ 11/18/2019

Estimating Entropy of Distributions in Constant Space

We consider the task of estimating the entropy of k-ary distributions fr...