DeepAI AI Chat
Log In Sign Up

A Comprehensive Study on Optimization Strategies for Gradient Descent In Deep Learning

by   Kaustubh Yadav, et al.

One of the most important parts of Artificial Neural Networks is minimizing the loss functions which tells us how good or bad our model is. To minimize these losses we need to tune the weights and biases. Also to calculate the minimum value of a function we need gradient. And to update our weights we need gradient descent. But there are some problems with regular gradient descent ie. it is quite slow and not that accurate. This article aims to give an introduction to optimization strategies to gradient descent. In addition, we shall also discuss the architecture of these algorithms and further optimization of Neural Networks in general


page 1

page 2

page 3

page 4


LocoProp: Enhancing BackProp via Local Loss Optimization

We study a local loss construction approach for optimizing neural networ...

Universality of Gradient Descent Neural Network Training

It has been observed that design choices of neural networks are often cr...

Approximation results for Gradient Descent trained Shallow Neural Networks in 1d

Two aspects of neural networks that have been extensively studied in the...

New Insights on Learning Rules for Hopfield Networks: Memory and Objective Function Minimisation

Hopfield neural networks are a possible basis for modelling associative ...

Learning without gradient descent encoded by the dynamics of a neurobiological model

The success of state-of-the-art machine learning is essentially all base...

Learning in Gated Neural Networks

Gating is a key feature in modern neural networks including LSTMs, GRUs ...