DeepAI AI Chat
Log In Sign Up

Asymptotic study of stochastic adaptive algorithm in non-convex landscape

by   Sébastien Gadat, et al.

This paper studies some asymptotic properties of adaptive algorithms widely used in optimization and machine learning, and among them Adagrad and Rmsprop, which are involved in most of the blackbox deep learning algorithms. Our setup is the non-convex landscape optimization point of view, we consider a one time scale parametrization and we consider the situation where these algorithms may be used or not with mini-batches. We adopt the point of view of stochastic algorithms and establish the almost sure convergence of these methods when using a decreasing step-size point of view towards the set of critical points of the target function. With a mild extra assumption on the noise, we also obtain the convergence towards the set of minimizer of the function. Along our study, we also obtain a "convergence rate" of the methods, in the vein of the works of <cit.>.


page 1

page 2

page 3

page 4


Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization

Although ADAM is a very popular algorithm for optimizing the weights of ...

Theoretical Study and Comparison of SPSA and RDSA Algorithms

Stochastic approximation (SA) algorithms are widely used in system optim...

First Order Methods for Geometric Optimization of Crystal Structures

The geometric optimisation of crystal structures is a procedure widely u...

On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization

This paper studies a class of adaptive gradient based momentum algorithm...

Dual perspective method for solving the point in a polygon problem

A novel method has been introduced to solve a point inclusion in a polyg...

Strong overall error analysis for the training of artificial neural networks via random initializations

Although deep learning based approximation algorithms have been applied ...

On Landscape of Lagrangian Functions and Stochastic Search for Constrained Nonconvex Optimization

We study constrained nonconvex optimization problems in machine learning...