Statistically Significant Stopping of Neural Network Training

by   J. K. Terry, et al.

The general approach taken when training deep learning classifiers is to save the parameters after every few iterations, train until either a human observer or a simple metric-based heuristic decides the network isn't learning anymore, and then backtrack and pick the saved parameters with the best validation accuracy. Simple methods are used to determine if a neural network isn't learning anymore because, as long as it's well after the optimal values are found, the condition doesn't impact the final accuracy of the model. However from a runtime perspective, this is of great significance to the many cases where numerous neural networks are trained simultaneously (e.g. hyper-parameter tuning). Motivated by this, we introduce a statistical significance test to determine if a neural network has stopped learning. This stopping criterion appears to represent a happy medium compared to other popular stopping criterions, achieving comparable accuracy to the criterions that achieve the highest final accuracies in 77 stop sooner do so with an appreciable loss to final accuracy. Additionally, we use this as the basis of a new learning rate scheduler, removing the need to manually choose learning rate schedules and acting as a quasi-line search, achieving superior or comparable empirical performance to existing methods.



There are no comments yet.


page 5

page 7


Gradient-only line searches to automatically determine learning rates for a variety of stochastic training algorithms

Gradient-only and probabilistic line searches have recently reintroduced...

Cyclical Learning Rates for Training Neural Networks

It is known that the learning rate is the most important hyper-parameter...

Neural Persistence: A Complexity Measure for Deep Neural Networks Using Algebraic Topology

While many approaches to make neural networks more fathomable have been ...

AutoLR: An Evolutionary Approach to Learning Rate Policies

The choice of a proper learning rate is paramount for good Artificial Ne...

Drawing early-bird tickets: Towards more efficient training of deep networks

(Frankle & Carbin, 2019) shows that there exist winning tickets (small b...

Dynamic learning rate using Mutual Information

This paper demonstrates dynamic hyper-parameter setting, for deep neural...

Sparse approximation in learning via neural ODEs

We consider the continuous-time, neural ordinary differential equation (...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.