Global Convergence and Geometric Characterization of Slow to Fast Weight Evolution in Neural Network Training for Classifying Linearly Non-Separable Data

02/28/2020

∙

In this paper, we study the dynamics of gradient descent in learning neural networks for classification problems. Unlike in existing works, we consider the linearly non-separable case where the training data of different classes lie in orthogonal subspaces. We show that when the network has sufficient (but not exceedingly large) number of neurons, (1) the corresponding minimization problem has a desirable landscape where all critical points are global minima with perfect classification; (2) gradient descent is guaranteed to converge to the global minima in this case. Moreover, we discovered a geometric condition on the network weights so that when it is satisfied, the weight evolution transitions from a slow phase of weight direction spreading to a fast phase of weight convergence. The geometric condition says that the convex hull of the weights projected on the unit sphere contains the origin.

READ FULL TEXT

Global Convergence and Geometric Characterization of Slow to Fast Weight Evolution in Neural Network Training for Classifying Linearly Non-Separable Data

A Geometric Approach of Gradient Descent Algorithms in Neural Networks

Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minima

Towards moderate overparameterization: global convergence guarantees for training shallow neural networks

Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations

Stationary Points of Shallow Neural Networks with Quadratic Activation Function

Learning ReLUs via Gradient Descent

Data-driven effective model shows a liquid-like deep learning

Global Convergence and Geometric Characterization of Slow to Fast Weight Evolution in Neural Network Training for Classifying Linearly Non-Separable Data

Related Research

A Geometric Approach of Gradient Descent Algorithms in Neural Networks

Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minima

Towards moderate overparameterization: global convergence guarantees for training shallow neural networks

Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations

Stationary Points of Shallow Neural Networks with Quadratic Activation Function

Learning ReLUs via Gradient Descent

Data-driven effective model shows a liquid-like deep learning