DeepAI AI Chat
Log In Sign Up

Halting Time is Predictable for Large Models: A Universality Property and Average-case Analysis

06/08/2020
by   Courtney Paquette, et al.
8

Average-case analysis computes the complexity of an algorithm averaged over all possible inputs. Compared to worst-case analysis, it is more representative of the typical behavior of an algorithm, but remains largely unexplored in optimization. One difficulty is that the analysis can depend on the probability distribution of the inputs to the model. However, we show that this is not the case for a class of large-scale problems trained with gradient descent including random least squares and one-hidden layer neural networks with random weights. In fact, the halting time exhibits a universality property: it is independent of the probability distribution. With this barrier for average-case analysis removed, we provide the first explicit average-case convergence rates showing a tighter complexity not captured by traditional worst-case analysis. Finally, numerical simulations suggest this universality property holds for a more general class of algorithms and problems.

READ FULL TEXT

page 1

page 2

page 3

page 4

07/26/2020

Distributional Analysis

In distributional or average-case analysis, the goal is to design an alg...
06/20/2022

Only Tails Matter: Average-Case Universality and Robustness in the Convex Regime

The recently developed average-case analysis of optimization methods all...
01/30/2018

Analysis of the Continued Logarithm Algorithm

The Continued Logarithm Algorithm - CL for short- introduced by Gosper i...
09/29/2021

Property Valuation Challenges for Properties Condemned for Environmental Public Upgrades

No property may be taken for public use without just compensation. The i...
02/08/2021

SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality

We propose a new framework, inspired by random matrix theory, for analyz...
05/22/2018

Gossip of Statistical Observations using Orthogonal Polynomials

Consider a network of agents connected by communication links, where eac...