Kernel Based Progressive Distillation for Adder Neural Networks

by   Yixing Xu, et al.

Adder Neural Networks (ANNs) which only contain additions bring us a new way of developing deep neural networks with low energy consumption. Unfortunately, there is an accuracy drop when replacing all convolution filters by adder filters. The main reason here is the optimization difficulty of ANNs using ℓ_1-norm, in which the estimation of gradient in back propagation is inaccurate. In this paper, we present a novel method for further improving the performance of ANNs without increasing the trainable parameters via a progressive kernel based knowledge distillation (PKKD) method. A convolutional neural network (CNN) with the same architecture is simultaneously initialized and trained as a teacher network, features and weights of ANN and CNN will be transformed to a new space to eliminate the accuracy drop. The similarity is conducted in a higher-dimensional space to disentangle the difference of their distributions using a kernel based method. Finally, the desired ANN is learned based on the information from both the ground-truth and teacher, progressively. The effectiveness of the proposed method for learning ANN with higher performance is then well-verified on several benchmarks. For instance, the ANN-50 trained using the proposed PKKD method obtains a 76.8% top-1 accuracy on ImageNet dataset, which is 0.6% higher than that of the ResNet-50.



There are no comments yet.


page 1

page 2

page 3

page 4


Progressive Label Distillation: Learning Input-Efficient Deep Neural Networks

Much of the focus in the area of knowledge distillation has been on dist...

HetConv: Heterogeneous Kernel-Based Convolutions for Deep CNNs

We present a novel deep learning architecture in which the convolution o...

Distilling Spikes: Knowledge Distillation in Spiking Neural Networks

Spiking Neural Networks (SNN) are energy-efficient computing architectur...

Follow Your Path: a Progressive Method for Knowledge Distillation

Deep neural networks often have a huge number of parameters, which posts...

Deep geometric knowledge distillation with graphs

In most cases deep learning architectures are trained disregarding the a...

Online Adversarial Distillation for Graph Neural Networks

Knowledge distillation has recently become a popular technique to improv...

Extracting Electron Scattering Cross Sections from Swarm Data using Deep Neural Networks

Electron-neutral scattering cross sections are fundamental quantities in...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.