AdaFamily: A family of Adam-like adaptive gradient methods

03/03/2022
by   Hannes Fassold, et al.
8

We propose AdaFamily, a novel method for training deep neural networks. It is a family of adaptive gradient methods and can be interpreted as sort of a blend of the optimization algorithms Adam, AdaBelief and AdaMomentum. We perform experiments on standard datasets for image classification, demonstrating that our proposed method outperforms these algorithms.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/26/2021

Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization

We introduce MADGRAD, a novel optimization method in the family of AdaGr...
research
07/28/2020

A Comparison of Optimization Algorithms for Deep Learning

In recent years, we have witnessed the rise of deep learning. Deep neura...
research
12/11/2020

The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks

Despite their overwhelming capacity to overfit, deep neural networks tra...
research
05/15/2021

On the Distributional Properties of Adaptive Gradients

Adaptive gradient methods have achieved remarkable success in training d...
research
12/06/2021

AdaSTE: An Adaptive Straight-Through Estimator to Train Binary Neural Networks

We propose a new algorithm for training deep neural networks (DNNs) with...
research
10/07/2021

G̅_mst:An Unbiased Stratified Statistic and a Fast Gradient Optimization Algorithm Based on It

-The fluctuation effect of gradient expectation and variance caused by p...
research
05/06/2023

Adam-family Methods for Nonsmooth Optimization with Convergence Guarantees

In this paper, we present a comprehensive study on the convergence prope...

Please sign up or login with your details

Forgot password? Click here to reset