Reusing Trained Layers of Convolutional Neural Networks to Shorten Hyperparameters Tuning Time

06/16/2020
by   Roberto L. Castro, et al.
0

Hyperparameters tuning is a time-consuming approach, particularly when the architecture of the neural network is decided as part of this process. For instance, in convolutional neural networks (CNNs), the selection of the number and the characteristics of the hidden (convolutional) layers may be decided. This implies that the search process involves the training of all these candidate network architectures. This paper describes a proposal to reuse the weights of hidden (convolutional) layers among different trainings to shorten this process. The rationale is that if a set of convolutional layers have been trained to solve a given problem, the weights calculated in this training may be useful when a new convolutional layer is added to the network architecture. This idea has been tested using the CIFAR-10 dataset, testing different CNNs architectures with up to 3 convolutional layers and up to 3 fully connected layers. The experiments compare the training time and the validation loss when reusing and not reusing convolutional layers. They confirm that this strategy reduces the training time while it even increases the accuracy of the resulting neural network. This finding opens up the future possibility of integrating this strategy in existing AutoML methods with the purpose of reducing the total search time.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/10/2019

Associative Convolutional Layers

Motivated by the necessity for parameter efficiency in distributed machi...
research
11/21/2019

Band-limited Training and Inference for Convolutional Neural Networks

The convolutional layers are core building blocks of neural network arch...
research
12/05/2020

Weight Update Skipping: Reducing Training Time for Artificial Neural Networks

Artificial Neural Networks (ANNs) are known as state-of-the-art techniqu...
research
03/17/2020

Hyperplane Arrangements of Trained ConvNets Are Biased

We investigate the geometric properties of the functions learned by trai...
research
03/23/2016

A guide to convolution arithmetic for deep learning

We introduce a guide to help deep learning practitioners understand and ...
research
12/07/2017

Distributed learning of CNNs on heterogeneous CPU/GPU architectures

Convolutional Neural Networks (CNNs) have shown to be powerful classific...
research
09/06/2023

Split-Boost Neural Networks

The calibration and training of a neural network is a complex and time-c...

Please sign up or login with your details

Forgot password? Click here to reset