Multirate Training of Neural Networks

06/20/2021
by   Tiffany Vlaar, et al.
0

We propose multirate training of neural networks: partitioning neural network parameters into "fast" and "slow" parts which are trained simultaneously using different learning rates. By choosing appropriate partitionings we can obtain large computational speed-ups for transfer learning tasks. We show that for various transfer learning applications in vision and NLP we can fine-tune deep neural networks in almost half the time, without reducing the generalization performance of the resulting model. We also discuss other splitting choices for the neural network parameters which are beneficial in enhancing generalization performance in settings where neural networks are trained from scratch. Finally, we propose an additional multirate technique which can learn different features present in the data by training the full network on different time scales simultaneously. The benefits of using this approach are illustrated for ResNet architectures on image data. Our paper unlocks the potential of using multirate techniques for neural network training and provides many starting points for future work in this area.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/20/2022

A Neural Network Based Method with Transfer Learning for Genetic Data Analysis

Transfer learning has emerged as a powerful technique in many applicatio...
research
05/31/2021

Persistent Homology Captures the Generalization of Neural Networks Without A Validation Set

The training of neural networks is usually monitored with a validation (...
research
02/09/2020

Improving Neural Network Learning Through Dual Variable Learning Rates

This paper introduces and evaluates a novel training method for neural n...
research
03/15/2020

Iterative training of neural networks for intra prediction

This paper presents an iterative training of neural networks for intra p...
research
08/03/2022

Maintaining Performance with Less Data

We propose a novel method for training a neural network for image classi...
research
01/24/2020

Performance of a Deep Neural Network at Detecting North Atlantic Right Whale Upcalls

Passive acoustics provides a powerful tool for monitoring the endangered...
research
01/07/2021

Transfer Learning Between Different Architectures Via Weights Injection

This work presents a naive algorithm for parameter transfer between diff...

Please sign up or login with your details

Forgot password? Click here to reset