Theory of Deep Learning IIb: Optimization Properties of SGD

01/07/2018

∙

In Theory IIb we characterize with a mix of theory and experiments the optimization of deep convolutional networks by Stochastic Gradient Descent. The main new result in this paper is theoretical and experimental evidence for the following conjecture about SGD: SGD concentrates in probability -- like the classical Langevin equation -- on large volume, "flat" minima, selecting flat minimizers which are with very high probability also global minimizers

READ FULL TEXT

Theory of Deep Learning IIb: Optimization Properties of SGD

A Diffusion Theory for Deep Learning Dynamics: Stochastic Gradient Descent Escapes From Sharp Minima Exponentially Fast

Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysis

Understanding Long Range Memory Effects in Deep Neural Networks

Quasi-potential theory for escape problem: Quantitative sharpness effect on SGD's escape from local minima

The Probabilistic Stability of Stochastic Gradient Descent

On regularization of gradient descent, layer imbalance and flat minima

On the SDEs and Scaling Rules for Adaptive Gradient Algorithms

Theory of Deep Learning IIb: Optimization Properties of SGD

Related Research

A Diffusion Theory for Deep Learning Dynamics: Stochastic Gradient Descent Escapes From Sharp Minima Exponentially Fast

Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysis

Understanding Long Range Memory Effects in Deep Neural Networks

Quasi-potential theory for escape problem: Quantitative sharpness effect on SGD's escape from local minima

The Probabilistic Stability of Stochastic Gradient Descent

On regularization of gradient descent, layer imbalance and flat minima

On the SDEs and Scaling Rules for Adaptive Gradient Algorithms