Event-Based Control for Online Training of Neural Networks

03/20/2020
by   Zilong Zhao, et al.
0

Convolutional Neural Network (CNN) has become the most used method for image classification tasks. During its training the learning rate and the gradient are two key factors to tune for influencing the convergence speed of the model. Usual learning rate strategies are time-based i.e. monotonous decay over time. Recent state-of-the-art techniques focus on adaptive gradient algorithms i.e. Adam and its versions. In this paper we consider an online learning scenario and we propose two Event-Based control loops to adjust the learning rate of a classical algorithm E (Exponential)/PD (Proportional Derivative)-Control. The first Event-Based control loop will be implemented to prevent sudden drop of the learning rate when the model is approaching the optimum. The second Event-Based control loop will decide, based on the learning speed, when to switch to the next data batch. Experimental evaluationis provided using two state-of-the-art machine learning image datasets (CIFAR-10 and CIFAR-100). Results show the Event-Based E/PD is better than the original algorithm (higher final accuracy, lower final loss value), and the Double-Event-BasedE/PD can accelerate the training process, save up to 67 state-of-the-art algorithms and even result in better performance.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/18/2019

Feedback Control for Online Training of Neural Networks

Convolutional neural networks (CNNs) are commonly used for image classif...
research
04/27/2019

Forget the Learning Rate, Decay Loss

In the usual deep neural network optimization process, the learning rate...
research
07/28/2019

ROAM: Recurrently Optimizing Tracking Model

Online updating a tracking model to adapt to object appearance variation...
research
03/23/2021

Evolving Learning Rate Optimizers for Deep Neural Networks

Artificial Neural Networks (ANNs) became popular due to their successful...
research
03/26/2019

Improving image classifiers for small datasets by learning rate adaptations

Our paper introduces an efficient combination of established techniques ...
research
01/06/2020

Self learning robot using real-time neural networks

With the advancements in high volume, low precision computational techno...
research
12/31/2019

Higher-order algorithms for nonlinearly parameterized adaptive control

A set of new adaptive control algorithms is presented. The algorithms ar...

Please sign up or login with your details

Forgot password? Click here to reset