Universal halting times in optimization and machine learning

11/19/2015
by   Levent Sagun, et al.
0

The authors present empirical distributions for the halting time (measured by the number of iterations to reach a given accuracy) of optimization algorithms applied to two random systems: spin glasses and deep learning. Given an algorithm, which we take to be both the optimization routine and the form of the random landscape, the fluctuations of the halting time follow a distribution that, after centering and scaling, remains unchanged even when the distribution on the landscape is changed. We observe two qualitative classes: A Gumbel-like distribution that appears in Google searches, human decision times, the QR eigenvalue algorithm and spin glasses, and a Gaussian-like distribution that appears in conjugate gradient method, deep network with MNIST input data and deep network with random input data. This empirical evidence suggests presence of a class of distributions for which the halting time is independent of the underlying distribution under some conditions.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/30/2019

Deep learning research landscape roadmap in a nutshell: past, present and future – Towards deep cortical learning

The past, present and future of deep learning is presented in this work....
research
07/16/2020

Data-driven effective model shows a liquid-like deep learning

Geometric structure of an optimization landscape is argued to be fundame...
research
11/22/2016

Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond

We look at the eigenvalues of the Hessian of a loss function before and ...
research
02/10/2017

Generative Mixture of Networks

A generative model based on training deep architectures is proposed. The...
research
01/16/2017

Datenqualität in Regressionsproblemen

Regression models are increasingly built using datasets which do not fol...
research
05/02/2017

Redundancy in active paths of deep networks: a random active path model

Deep learning has become a powerful and popular tool for a variety of ma...
research
09/15/2023

Do Random and Chaotic Sequences Really Cause Different PSO Performance? Further Results

Empirical results show that PSO performance may be different if using ei...

Please sign up or login with your details

Forgot password? Click here to reset