Minimization of Stochastic First-order Oracle Complexity of Adaptive Methods for Nonconvex Optimization

by   Hideaki Iiduka, et al.

Numerical evaluations have definitively shown that, for deep learning optimizers such as stochastic gradient descent, momentum, and adaptive methods, the number of steps needed to train a deep neural network halves for each doubling of the batch size and that there is a region of diminishing returns beyond the critical batch size. In this paper, we determine the actual critical batch size by using the global minimizer of the stochastic first-order oracle (SFO) complexity of the optimizer. To prove the existence of the actual critical batch size, we set the lower and upper bounds of the SFO complexity and prove that there exist critical batch sizes in the sense of minimizing the lower and upper bounds. This proof implies that, if the SFO complexity fits the lower and upper bounds, then the existence of these critical batch sizes demonstrates the existence of the actual critical batch size. We also discuss the conditions needed for the SFO complexity to fit the lower and upper bounds and provide numerical results that support our theoretical results.


page 1

page 2

page 3

page 4


Critical Bach Size Minimizes Stochastic First-Order Oracle Complexity of Deep Learning Optimizer using Hyperparameters Close to One

Practical results have shown that deep learning optimizers using small c...

The Number of Steps Needed for Nonconvex Optimization of a Deep Learning Optimizer is a Rational Function of Batch Size

Recently, convergence as well as convergence rate analyses of deep learn...

Bounds for the VC Dimension of 1NN Prototype Sets

In Statistical Learning, the Vapnik-Chervonenkis (VC) dimension is an im...

Select without Fear: Almost All Mini-Batch Schedules Generalize Optimally

We establish matching upper and lower generalization error bounds for mi...

A survey on stability measure of networks

In this paper, we discuss tenacity and its properties in the stability c...

When are epsilon-nets small?

In many interesting situations the size of epsilon-nets depends only on ...

Please sign up or login with your details

Forgot password? Click here to reset