Statistical-Query Lower Bounds via Functional Gradients

06/29/2020
by   Surbhi Goel, et al.
0

We give the first statistical-query lower bounds for agnostically learning any non-polynomial activation with respect to Gaussian marginals (e.g., ReLU, sigmoid, sign). For the specific problem of ReLU regression (equivalently, agnostically learning a ReLU), we show that any statistical-query algorithm with tolerance n^-Θ(ϵ^-1/2) must use at least 2^n^cϵ queries for some constant c > 0, where n is the dimension and ϵ is the accuracy parameter. Our results rule out general (as opposed to correlational) SQ learning algorithms, which is unusual for real-valued learning problems. Our techniques involve a gradient boosting procedure for "amplifying" recent lower bounds due to Diakonikolas et al. (COLT 2020) and Goel et al. (ICML 2020) on the SQ dimension of functions computed by two-layer neural networks. The crucial new ingredient is the use of a nonstandard convex functional during the boosting procedure. This also yields a best-possible reduction between two commonly studied models of learning: agnostic learning and probabilistic concepts.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/10/2022

Hardness of Noise-Free Learning for Two-Hidden-Layer Neural Networks

We give superpolynomial statistical query (SQ) lower bounds for learning...
research
02/10/2022

Near-Optimal Statistical Query Lower Bounds for Agnostically Learning Intersections of Halfspaces with Gaussian Marginals

We consider the well-studied problem of learning intersections of halfsp...
research
06/22/2020

Algorithms and SQ Lower Bounds for PAC Learning One-Hidden-Layer ReLU Networks

We study the problem of PAC learning one-hidden-layer ReLU networks with...
research
06/05/2023

On the Role of Entanglement and Statistics in Learning

In this work we make progress in understanding the relationship between ...
research
08/07/2016

A General Characterization of the Statistical Query Complexity

Statistical query (SQ) algorithms are algorithms that have access to an ...
research
02/08/2021

The Optimality of Polynomial Regression for Agnostic Learning under Gaussian Marginals

We study the problem of agnostic learning under the Gaussian distributio...
research
05/31/2022

Learning (Very) Simple Generative Models Is Hard

Motivated by the recent empirical successes of deep generative models, w...

Please sign up or login with your details

Forgot password? Click here to reset