-
Understanding and Detecting Convergence for Stochastic Gradient Descent with Momentum
Convergence detection of iterative stochastic optimization methods is of...
read it
-
A Probabilistically Motivated Learning Rate Adaptation for Stochastic Optimization
Machine learning practitioners invest significant manual and computation...
read it
-
Using Statistics to Automate Stochastic Optimization
Despite the development of numerous adaptive optimizers, tuning the lear...
read it
-
Training Deep Networks without Learning Rates Through Coin Betting
Deep learning methods achieve state-of-the-art performance in many appli...
read it
-
Data augmentation as stochastic optimization
We present a theoretical framework recasting data augmentation as stocha...
read it
-
Improving reinforcement learning algorithms: towards optimal learning rate policies
This paper investigates to what extent we can improve reinforcement lear...
read it
-
A comparison of learning rate selection methods in generalized Bayesian inference
Generalized Bayes posterior distributions are formed by putting a fracti...
read it
Robust Learning Rate Selection for Stochastic Optimization via Splitting Diagnostic
This paper proposes SplitSGD, a new stochastic optimization algorithm with a dynamic learning rate selection rule. This procedure decreases the learning rate for better adaptation to the local geometry of the objective function whenever a stationary phase is detected, that is, the iterates are likely to bounce around a vicinity of a local minimum. The detection is performed by splitting the single thread into two and using the inner products of the gradients from the two threads as a measure of stationarity. This learning rate selection is provably valid, robust to initial parameters, easy-to-implement, and essentially does not incur additional computational cost. Finally, we illustrate the robust convergence properties of SplitSGD through extensive experiments.
READ FULL TEXT
Comments
There are no comments yet.