Threshold Phenomena in Learning Halfspaces with Massart Noise

by   Ilias Diakonikolas, et al.

We study the problem of PAC learning halfspaces on ℝ^d with Massart noise under Gaussian marginals. In the Massart noise model, an adversary is allowed to flip the label of each point 𝐱 with probability η(𝐱) ≤η, for some parameter η∈ [0,1/2]. The goal of the learner is to output a hypothesis with missclassification error opt + ϵ, where opt is the error of the target halfspace. Prior work studied this problem assuming that the target halfspace is homogeneous and that the parameter η is strictly smaller than 1/2. We explore how the complexity of the problem changes when either of these assumptions is removed, establishing the following threshold phenomena: For η = 1/2, we prove a lower bound of d^Ω (log(1/ϵ)) on the complexity of any Statistical Query (SQ) algorithm for the problem, which holds even for homogeneous halfspaces. On the positive side, we give a new learning algorithm for arbitrary halfspaces in this regime with sample complexity and running time O_ϵ(1) d^O(log(1/ϵ)). For η <1/2, we establish a lower bound of d^Ω(log(1/γ)) on the SQ complexity of the problem, where γ = max{ϵ, min{𝐏𝐫[f(𝐱) = 1], 𝐏𝐫[f(𝐱) = -1]}} and f is the target halfspace. In particular, this implies an SQ lower bound of d^Ω (log(1/ϵ) ) for learning arbitrary Massart halfspaces (even for small constant η). We complement this lower bound with a new learning algorithm for this regime with sample complexity and runtime d^O_η(log(1/γ))poly(1/ϵ). Taken together, our results qualitatively characterize the complexity of learning halfspaces in the Massart model.


page 1

page 2

page 3

page 4


On the Power of Localized Perceptron for Label-Optimal Learning of Halfspaces with Adversarial Noise

We study online active learning of homogeneous halfspaces in ℝ^d with ad...

Optimal learning of quantum Hamiltonians from high-temperature Gibbs states

We study the problem of learning a Hamiltonian H to precision ε, supposi...

Provable Lifelong Learning of Representations

In lifelong learning, the tasks (or classes) to be learned arrive sequen...

Cryptographic Hardness of Learning Halfspaces with Massart Noise

We study the complexity of PAC learning halfspaces in the presence of Ma...

Query Complexity of Bayesian Private Learning

We study the query complexity of Bayesian Private Learning: a learner wi...

Sample Complexity Bounds for Robustly Learning Decision Lists against Evasion Attacks

A fundamental problem in adversarial machine learning is to quantify how...

Memory-Sample Lower Bounds for Learning Parity with Noise

In this work, we show, for the well-studied problem of learning parity u...