Learning ReLUs via Gradient Descent

05/10/2017
by   Mahdi Soltanolkotabi, et al.
0

In this paper we study the problem of learning Rectified Linear Units (ReLUs) which are functions of the form max(0,<w,x>) with w denoting the weight vector. We study this problem in the high-dimensional regime where the number of observations are fewer than the dimension of the weight vector. We assume that the weight vector belongs to some closed set (convex or nonconvex) which captures known side-information about its structure. We focus on the realizable model where the inputs are chosen i.i.d. from a Gaussian distribution and the labels are generated according to a planted weight vector. We show that projected gradient descent, when initialization at 0, converges at a linear rate to the planted model with a number of samples that is optimal up to numerical constants. Our results on the dynamics of convergence of these very shallow neural nets may provide some insights towards understanding the dynamics of deeper architectures.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/16/2017

Theoretical insights into the optimization landscape of over-parameterized shallow neural networks

In this paper we study the problem of learning a shallow artificial neur...
research
04/06/2018

Reconstructing Point Sets from Distance Distributions

We study the problem of reconstructing the locations u_i of a set of po...
research
06/23/2016

An Approach to Stable Gradient Descent Adaptation of Higher-Order Neural Units

Stability evaluation of a weight-update system of higher-order neural un...
research
10/23/2016

Fast and Reliable Parameter Estimation from Nonlinear Observations

In this paper we study the problem of recovering a structured but unknow...
research
01/19/2019

Fitting ReLUs via SGD and Quantized SGD

In this paper we focus on the problem of finding the optimal weights of ...
research
07/03/2019

Quickly Finding the Best Linear Model in High Dimensions

We study the problem of finding the best linear model that can minimize ...

Please sign up or login with your details

Forgot password? Click here to reset