Threshold Phenomena in Learning Halfspaces with Massart Noise

08/19/2021
by   Ilias Diakonikolas, et al.
11

We study the problem of PAC learning halfspaces on ℝ^d with Massart noise under Gaussian marginals. In the Massart noise model, an adversary is allowed to flip the label of each point 𝐱 with probability η(𝐱) ≤η, for some parameter η∈ [0,1/2]. The goal of the learner is to output a hypothesis with missclassification error opt + ϵ, where opt is the error of the target halfspace. Prior work studied this problem assuming that the target halfspace is homogeneous and that the parameter η is strictly smaller than 1/2. We explore how the complexity of the problem changes when either of these assumptions is removed, establishing the following threshold phenomena: For η = 1/2, we prove a lower bound of d^Ω (log(1/ϵ)) on the complexity of any Statistical Query (SQ) algorithm for the problem, which holds even for homogeneous halfspaces. On the positive side, we give a new learning algorithm for arbitrary halfspaces in this regime with sample complexity and running time O_ϵ(1) d^O(log(1/ϵ)). For η <1/2, we establish a lower bound of d^Ω(log(1/γ)) on the SQ complexity of the problem, where γ = max{ϵ, min{𝐏𝐫[f(𝐱) = 1], 𝐏𝐫[f(𝐱) = -1]}} and f is the target halfspace. In particular, this implies an SQ lower bound of d^Ω (log(1/ϵ) ) for learning arbitrary Massart halfspaces (even for small constant η). We complement this lower bound with a new learning algorithm for this regime with sample complexity and runtime d^O_η(log(1/γ))poly(1/ϵ). Taken together, our results qualitatively characterize the complexity of learning halfspaces in the Massart model.

READ FULL TEXT

page 1

page 2

page 3

page 4

12/19/2020

On the Power of Localized Perceptron for Label-Optimal Learning of Halfspaces with Adversarial Noise

We study online active learning of homogeneous halfspaces in ℝ^d with ad...
08/10/2021

Optimal learning of quantum Hamiltonians from high-temperature Gibbs states

We study the problem of learning a Hamiltonian H to precision ε, supposi...
10/27/2021

Provable Lifelong Learning of Representations

In lifelong learning, the tasks (or classes) to be learned arrive sequen...
07/28/2022

Cryptographic Hardness of Learning Halfspaces with Massart Noise

We study the complexity of PAC learning halfspaces in the presence of Ma...
11/15/2019

Query Complexity of Bayesian Private Learning

We study the query complexity of Bayesian Private Learning: a learner wi...
05/12/2022

Sample Complexity Bounds for Robustly Learning Decision Lists against Evasion Attacks

A fundamental problem in adversarial machine learning is to quantify how...
07/05/2021

Memory-Sample Lower Bounds for Learning Parity with Noise

In this work, we show, for the well-studied problem of learning parity u...