Dynamic learning rate using Mutual Information

05/18/2018
by   Shrihari Vasudevan, et al.
0

This paper demonstrates dynamic hyper-parameter setting, for deep neural network training, using Mutual Information (MI). The specific hyper-parameter studied in this paper is the learning rate. MI between the output layer and true outcomes is used to dynamically set the learning rate of the network through the training cycle; the idea is also extended to layer-wise setting of learning rate. Two approaches are demonstrated - tracking relative change in mutual information and, additionally tracking its value relative to a reference measure. The paper does not attempt to recommend a specific learning rate policy. Experiments demonstrate that mutual information may be effectively used to dynamically set learning rate and achieve competitive to better outcomes in competitive to better time.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/08/2020

AutoLR: An Evolutionary Approach to Learning Rate Policies

The choice of a proper learning rate is paramount for good Artificial Ne...
research
12/08/2020

Mutual Information Decay Curves and Hyper-Parameter Grid Search Design for Recurrent Neural Architectures

We present an approach to design the grid searches for hyper-parameter o...
research
04/27/2019

Forget the Learning Rate, Decay Loss

In the usual deep neural network optimization process, the learning rate...
research
11/21/2022

Self-Adaptive, Dynamic, Integrated Statistical and Information Theory Learning

The paper analyses and serves with a positioning of various error measur...
research
06/24/2022

Mutual-Information Based Optimal Experimental Design for Hyperpolarized ^13C-Pyruvate MRI

A key parameter of interest recovered from hyperpolarized (HP) MRI measu...
research
06/16/2021

To Raise or Not To Raise: The Autonomous Learning Rate Question

There is a parameter ubiquitous throughout the deep learning world: lear...
research
11/01/2022

Higher-order mutual information reveals synergistic sub-networks for multi-neuron importance

Quantifying which neurons are important with respect to the classificati...

Please sign up or login with your details

Forgot password? Click here to reset