Associative Convolutional Layers

06/10/2019
by   Hamed Omidvar, et al.
11

Motivated by the necessity for parameter efficiency in distributed machine learning and AI-enabled edge devices, we provide a general and easy to implement method for significantly reducing the number of parameters of Convolutional Neural Networks (CNNs), during both the training and inference phases. We introduce a simple auxiliary neural network which can generate the convolutional filters of any CNN architecture from a low dimensional latent space. This auxiliary neural network, which we call "Convolutional Slice Generator" (CSG), is unique to the network and provides the association between its convolutional layers. During the training of the CNN, instead of training the filters of the convolutional layers, only the parameters of the CSG and their corresponding `code vectors' are trained. This results in a significant reduction of the number of parameters due to the fact that the CNN can be fully represented using only the parameters of the CSG, the code vectors, the fully connected layers, and the architecture of the CNN. To show the capability of our method, we apply it to ResNet and DenseNet architectures, using the CIFAR-10 dataset without any hyper-parameter tuning. Experiments show that our approach, even when applied to already compressed and efficient CNNs such as DenseNet-BC, significantly reduces the number of network parameters. In two models based on DenseNet-BC with ≈ 2× reduction in one of them we had a slight improvement in accuracy and in another one, with ≈ 2× reduction the change in accuracy is negligible. In case of ResNet-56, ≈ 2.5× reduction leads to an accuracy loss within 1%.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/16/2020

Reusing Trained Layers of Convolutional Neural Networks to Shorten Hyperparameters Tuning Time

Hyperparameters tuning is a time-consuming approach, particularly when t...
research
06/10/2020

Better Together: Resnet-50 accuracy with 13x fewer parameters and at 3x speed

Recent research on compressing deep neural networks has focused on reduc...
research
02/27/2019

Reducing Artificial Neural Network Complexity: A Case Study on Exoplanet Detection

Despite their successes in the field of self-learning AI, Convolutional ...
research
09/16/2019

A few filters are enough: Convolutional Neural Network for P300 Detection

In this paper, we aim to provide elements to contribute to the discussio...
research
12/21/2013

Spectral Networks and Locally Connected Networks on Graphs

Convolutional Neural Networks are extremely efficient architectures in i...
research
01/19/2020

Towards More Efficient and Effective Inference: The Joint Decision of Multi-Participants

Existing approaches to improve the performances of convolutional neural ...
research
12/14/2016

Beam Search for Learning a Deep Convolutional Neural Network of 3D Shapes

This paper addresses 3D shape recognition. Recent work typically represe...

Please sign up or login with your details

Forgot password? Click here to reset