PAL: A fast DNN optimization method based on curvature information

03/28/2019
by   Maximus Mutschler, et al.
0

We present a novel optimizer for deep neural networks that combines the ideas of Netwon's method and line search to efficiently compute and utilize curvature information. Our work is based on empirical observation suggesting that the loss function can be approximated by a parabola in negative gradient direction. Due to this approximation, we are able to perform a variable and loss function dependent parameter update by jumping directly into the minimum of the approximated parabola. To evaluate our optimizer, we performed multiple comprehensive hyperparameter grid searches for which we trained more than 20000 networks in total. We can show that PAL outperforms RMSPROP, and can outperform gradient descent with momentum and ADAM on large-scale high-dimensional machine learning problems. Furthermore, PAL requires up to 52.2 PyTorch and TensorFlow implementations are provided at https://github.com/cogsys-tuebingen/PAL.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/26/2021

Curvature Injected Adaptive Momentum Optimizer for Convolutional Neural Networks

In this paper, we propose a new approach, hereafter referred as AdaInjec...
research
02/29/2020

TAdam: A Robust Stochastic Gradient Optimizer

Machine learning algorithms aim to find patterns from observations, whic...
research
02/14/2018

L4: Practical loss-based stepsize adaptation for deep learning

We propose a stepsize adaptation scheme for stochastic gradient descent....
research
04/03/2020

Gradient Centralization: A New Optimization Technique for Deep Neural Networks

Optimization techniques are of great importance to effectively and effic...
research
01/02/2020

A Loss-Function for Causal Machine-Learning

Causal machine-learning is about predicting the net-effect (true-lift) o...
research
10/01/2021

Update in Unit Gradient

In Machine Learning, optimization mostly has been done by using a gradie...

Please sign up or login with your details

Forgot password? Click here to reset