DeepAI AI Chat
Log In Sign Up

Training Deep Spiking Auto-encoders without Bursting or Dying Neurons through Regularization

by   Justus F. Hübotter, et al.

Spiking neural networks are a promising approach towards next-generation models of the brain in computational neuroscience. Moreover, compared to classic artificial neural networks, they could serve as an energy-efficient deployment of AI by enabling fast computation in specialized neuromorphic hardware. However, training deep spiking neural networks, especially in an unsupervised manner, is challenging and the performance of a spiking model is significantly hindered by dead or bursting neurons. Here, we apply end-to-end learning with membrane potential-based backpropagation to a spiking convolutional auto-encoder with multiple trainable layers of leaky integrate-and-fire neurons. We propose bio-inspired regularization methods to control the spike density in latent representations. In the experiments, we show that applying regularization on membrane potential and spiking output successfully avoids both dead and bursting neurons and significantly decreases the reconstruction error of the spiking auto-encoder. Training regularized networks on the MNIST dataset yields image reconstruction quality comparable to non-spiking baseline models (deterministic and variational auto-encoder) and indicates improvement upon earlier approaches. Importantly, we show that, unlike the variational auto-encoder, the spiking latent representations display structure associated with the image class.


page 7

page 16

page 17

page 20

page 21


Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks

The emergence of brain-inspired neuromorphic computing as a paradigm for...

Deep Auto-encoder with Neural Response

Artificial intelligence and neuroscience are deeply interactive. Artific...

AutoEncoder by Forest

Auto-encoding is an important task which is typically realized by deep n...

Toward an Over-parameterized Direct-Fit Model of Visual Perception

In this paper, we revisit the problem of computational modeling of simpl...

Spiking Variational Graph Auto-Encoders for Efficient Graph Representation Learning

Graph representation learning is a fundamental research issue and benefi...

Unsupervised Regenerative Learning of Hierarchical Features in Spiking Deep Networks for Object Recognition

We present a spike-based unsupervised regenerative learning scheme to tr...

LFADS - Latent Factor Analysis via Dynamical Systems

Neuroscience is experiencing a data revolution in which many hundreds or...