Training Energy-Efficient Deep Spiking Neural Networks with Time-to-First-Spike Coding

06/04/2021
by   Seongsik Park, et al.
0

The tremendous energy consumption of deep neural networks (DNNs) has become a serious problem in deep learning. Spiking neural networks (SNNs), which mimic the operations in the human brain, have been studied as prominent energy-efficient neural networks. Due to their event-driven and spatiotemporally sparse operations, SNNs show possibilities for energy-efficient processing. To unlock their potential, deep SNNs have adopted temporal coding such as time-to-first-spike (TTFS)coding, which represents the information between neurons by the first spike time. With TTFS coding, each neuron generates one spike at most, which leads to a significant improvement in energy efficiency. Several studies have successfully introduced TTFS coding in deep SNNs, but they showed restricted efficiency improvement owing to the lack of consideration for efficiency during training. To address the aforementioned issue, this paper presents training methods for energy-efficient deep SNNs with TTFS coding. We introduce a surrogate DNN model to train the deep SNN in a feasible time and analyze the effect of the temporal kernel on training performance and efficiency. Based on the investigation, we propose stochastically relaxed activation and initial value-based regularization for the temporal kernel parameters. In addition, to reduce the number of spikes even further, we present temporal kernel-aware batch normalization. With the proposed methods, we could achieve comparable training results with significantly reduced spikes, which could lead to energy-efficient deep SNNs.

READ FULL TEXT

page 3

page 4

page 5

page 8

page 9

page 10

page 11

page 15

03/26/2020

T2FSNN: Deep Spiking Neural Networks with Time-to-first-spike Coding

Spiking neural networks (SNNs) have gained considerable interest due to ...
04/22/2021

Noise-Robust Deep Spiking Neural Networks with Temporal Information

Spiking neural networks (SNNs) have emerged as energy-efficient neural n...
03/17/2020

SiamSNN: Spike-based Siamese Network for Energy-Efficient and Real-time Object Tracking

Although deep neural networks (DNNs) have achieved fantastic success in ...
01/30/2022

AutoSNN: Towards Energy-Efficient Spiking Neural Networks

Spiking neural networks (SNNs) that mimic information transmission in th...
01/31/2022

Rate Coding or Direct Coding: Which One is Better for Accurate, Robust, and Energy-efficient Spiking Neural Networks?

Recent Spiking Neural Networks (SNNs) works focus on an image classifica...
05/26/2022

Learning in Feedback-driven Recurrent Spiking Neural Networks using full-FORCE Training

Feedback-driven recurrent spiking neural networks (RSNNs) are powerful c...
02/10/2020

A Spike in Performance: Training Hybrid-Spiking Neural Networks with Quantized Activation Functions

The machine learning community has become increasingly interested in the...