Optimistic Adaptive Acceleration for Optimization

03/04/2019
by   Jun-Kun Wang, et al.
12

We consider a new variant of AMSGrad. AMSGrad RKK18 is a popular adaptive gradient based optimization algorithm that is widely used in training deep neural networks. Our new variant of the algorithm assumes that mini-batch gradients in consecutive iterations have some underlying structure, which makes the gradients sequentially predictable. By exploiting the predictability and some ideas from the field of Optimistic Online learning, the new algorithm can accelerate the convergence and enjoy a tighter regret bound. We conduct experiments on training various neural networks on several datasets to show that the proposed method speeds up the convergence in practice.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/19/2018

Nostalgic Adam: Weighing more of the past gradients when designing the adaptive learning rate

First-order optimization methods have been playing a prominent role in d...
research
01/01/2021

Adam revisited: a weighted past gradients perspective

Adaptive learning rate methods have been successfully applied in many fi...
research
05/31/2016

Adaptive Learning Rate via Covariance Matrix Based Preconditioning for Deep Neural Networks

Adaptive learning rate algorithms such as RMSProp are widely used for tr...
research
06/17/2017

Variants of RMSProp and Adagrad with Logarithmic Regret Bounds

Adaptive gradient methods have become recently very popular, in particul...
research
04/07/2019

On the convergence proof of AMSGrad and a new version

The adaptive moment estimation algorithm Adam (Kingma and Ba, ICLR 2015)...
research
12/06/2021

AdaSTE: An Adaptive Straight-Through Estimator to Train Binary Neural Networks

We propose a new algorithm for training deep neural networks (DNNs) with...
research
01/09/2023

Balance is Essence: Accelerating Sparse Training via Adaptive Gradient Correction

Despite impressive performance on a wide variety of tasks, deep neural n...

Please sign up or login with your details

Forgot password? Click here to reset