Energy-efficient Knowledge Distillation for Spiking Neural Networks

06/14/2021
by   Dongjin Lee, et al.
0

Spiking neural networks (SNNs) have been gaining interest as energy-efficient alternatives of conventional artificial neural networks (ANNs) due to their event-driven computation. Considering the future deployment of SNN models to constrained neuromorphic devices, many studies have applied techniques originally used for ANN model compression, such as network quantization, pruning, and knowledge distillation, to SNNs. Among them, existing works on knowledge distillation reported accuracy improvements of student SNN model. However, analysis on energy efficiency, which is also an important feature of SNN, was absent. In this paper, we thoroughly analyze the performance of the distilled SNN model in terms of accuracy and energy efficiency. In the process, we observe a substantial increase in the number of spikes, leading to energy inefficiency, when using the conventional knowledge distillation methods. Based on this analysis, to achieve energy efficiency, we propose a novel knowledge distillation method with heterogeneous temperature parameters. We evaluate our method on two different datasets and show that the resulting SNN student satisfies both accuracy improvement and reduction of the number of spikes. On MNIST dataset, our proposed student SNN achieves up to 0.09 and produces 65 conventional knowledge distillation method. We also compare the results with other SNN compression techniques and training methods.

READ FULL TEXT

page 2

page 3

page 8

page 9

page 10

page 11

page 13

page 14

05/01/2020

Distilling Spikes: Knowledge Distillation in Spiking Neural Networks

Spiking Neural Networks (SNN) are energy-efficient computing architectur...
12/17/2020

Computation-Efficient Knowledge Distillation via Uncertainty-Aware Mixup

Knowledge distillation, which involves extracting the "dark knowledge" f...
10/24/2018

HAKD: Hardware Aware Knowledge Distillation

Despite recent developments, deploying deep neural networks on resource ...
08/29/2021

Lipschitz Continuity Guided Knowledge Distillation

Knowledge distillation has become one of the most important model compre...
06/03/2020

Stochastic-based Neural Network hardware acceleration for an efficient ligand-based virtual screening

Artificial Neural Networks (ANN) have been popularized in many science a...
09/09/2020

On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective

To put a state-of-the-art neural network to practical use, it is necessa...
05/11/2021

Pruning of Deep Spiking Neural Networks through Gradient Rewiring

Spiking Neural Networks (SNNs) have been attached great importance due t...