An Elementary Approach to Convergence Guarantees of Optimization Algorithms for Deep Networks

02/20/2020
by   Vincent Roulet, et al.
0

We present an approach to obtain convergence guarantees of optimization algorithms for deep networks based on elementary arguments and computations. The convergence analysis revolves around the analytical and computational structures of optimization oracles central to the implementation of deep networks in machine learning software. We provide a systematic way to compute estimates of the smoothness constants that govern the convergence behavior of first-order optimization algorithms used to train deep networks. A diverse set of example components and architectures arising in modern deep networks intersperse the exposition to illustrate the approach.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/11/2019

Analysis of Optimization Algorithms via Sum-of-Squares

In this work, we introduce a new framework for unifying and systematizin...
research
07/28/2020

A Comparison of Optimization Algorithms for Deep Learning

In recent years, we have witnessed the rise of deep learning. Deep neura...
research
07/30/2018

Faster Convergence & Generalization in DNNs

Deep neural networks have gained tremendous popularity in last few years...
research
04/01/2021

Fast Jacobian-Vector Product for Deep Networks

Jacobian-vector products (JVPs) form the backbone of many recent develop...
research
05/27/2021

How saccadic vision might help with theinterpretability of deep networks

We describe how some problems (interpretability,lack of object-orientedn...
research
06/16/2021

On the proper role of linguistically-oriented deep net analysis in linguistic theorizing

A lively research field has recently emerged that uses experimental meth...
research
11/02/2018

Invertible Residual Networks

Reversible deep networks provide useful theoretical guarantees and have ...

Please sign up or login with your details

Forgot password? Click here to reset