Learning Single-Index Models with Shallow Neural Networks

10/27/2022
by   Alberto Bietti, et al.
0

Single-index models are a class of functions given by an unknown univariate “link” function applied to an unknown one-dimensional projection of the input. These models are particularly relevant in high dimension, when the data might present low-dimensional structure that learning algorithms should adapt to. While several statistical aspects of this model, such as the sample complexity of recovering the relevant (one-dimensional) subspace, are well-understood, they rely on tailored algorithms that exploit the specific structure of the target function. In this work, we introduce a natural class of shallow neural networks and study its ability to learn single-index models via gradient flow. More precisely, we consider shallow networks in which biases of the neurons are frozen at random initialization. We show that the corresponding optimization landscape is benign, which in turn leads to generalization guarantees that match the near-optimal sample complexity of dedicated semi-parametric methods.

READ FULL TEXT
research
07/28/2023

On Single Index Models beyond Gaussian Data

Sparse high-dimensional functions have arisen as a rich framework to stu...
research
09/29/2022

Neural Networks Efficiently Learn Low-Dimensional Representations with SGD

We study the problem of training a two-layer neural network (NN) of arbi...
research
05/11/2023

Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networks

One of the central questions in the theory of deep learning is to unders...
research
04/28/2020

Learning Polynomials of Few Relevant Dimensions

Polynomial regression is a basic primitive in learning and statistics. I...
research
08/17/2022

Shallow neural network representation of polynomials

We show that d-variate polynomials of degree R can be represented on [0,...
research
06/30/2015

Learning Single Index Models in High Dimensions

Single Index Models (SIMs) are simple yet flexible semi-parametric model...
research
09/07/2023

Gradient-Based Feature Learning under Structured Data

Recent works have demonstrated that the sample complexity of gradient-ba...

Please sign up or login with your details

Forgot password? Click here to reset