Stochastic modified equations for the asynchronous stochastic gradient descent

05/21/2018
by   Jing An, et al.
0

We propose a stochastic modified equations (SME) for modeling the asynchronous stochastic gradient descent (ASGD) algorithms. The resulting SME of Langevin type extracts more information about the ASGD dynamics and elucidates the relationship between different types of stochastic gradient algorithms. We show the convergence of ASGD to the SME in the continuous time limit, as well as the SME's precise prediction to the trajectories of ASGD with various forcing terms. As an application of the SME, we propose an optimal mini-batching strategy for ASGD via solving the optimal control problem of the associated SME.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/24/2020

A Sharp Convergence Rate for the Asynchronous Stochastic Gradient Descent

We give a sharp convergence rate for the asynchronous stochastic gradien...
research
06/07/2021

Learning Stochastic Optimal Policies via Gradient Descent

We systematically develop a learning-based treatment of stochastic optim...
research
11/19/2015

Stochastic modified equations and adaptive stochastic gradient algorithms

We develop the method of stochastic modified equations (SME), in which s...
research
01/23/2013

Learning Finite-State Controllers for Partially Observable Environments

Reactive (memoryless) policies are sufficient in completely observable M...
research
04/04/2018

GoSGD: Distributed Optimization for Deep Learning with Gossip Exchange

We address the issue of speeding up the training of convolutional neural...
research
08/11/2023

The Stochastic Steepest Descent Method for Robust Optimization in Banach Spaces

Stochastic gradient methods have been a popular and powerful choice of o...
research
06/20/2022

Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisation

Understanding the implicit bias of training algorithms is of crucial imp...

Please sign up or login with your details

Forgot password? Click here to reset