Improving Layer-wise Adaptive Rate Methods using Trust Ratio Clipping

by   Jeffrey Fong, et al.
National University of Singapore

Training neural networks with large batch is of fundamental significance to deep learning. Large batch training remarkably reduces the amount of training time but has difficulties in maintaining accuracy. Recent works have put forward optimization methods such as LARS and LAMB to tackle this issue through adaptive layer-wise optimization using trust ratios. Though prevailing, such methods are observed to still suffer from unstable and extreme trust ratios which degrades performance. In this paper, we propose a new variant of LAMB, called LAMBC, which employs trust ratio clipping to stabilize its magnitude and prevent extreme values. We conducted experiments on image classification tasks such as ImageNet and CIFAR-10 and our empirical results demonstrate promising improvements across different batch sizes.


page 1

page 2

page 3

page 4


Large Batch Training Does Not Need Warmup

Training deep neural networks using a large batch size has shown promisi...

Large Batch Training of Convolutional Networks

A common way to speed up training of large convolutional networks is to ...

A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes

Recently the LARS and LAMB optimizers have been proposed for training ne...

Layer-wise Adaptive Step-Sizes for Stochastic First-Order Methods for Deep Learning

We propose a new per-layer adaptive step-size procedure for stochastic f...

Adaptive Gradient Methods with Dynamic Bound of Learning Rate

Adaptive optimization methods such as AdaGrad, RMSprop and Adam have bee...

Improving training of deep neural networks via Singular Value Bounding

Deep learning methods achieve great success recently on many computer vi...

Adaptive Fractional Dilated Convolution Network for Image Aesthetics Assessment

To leverage deep learning for image aesthetics assessment, one critical ...

Please sign up or login with your details

Forgot password? Click here to reset