Online Spatio-Temporal Learning in Deep Neural Networks

by   Thomas Bohnstingl, et al.

Biological neural networks are equipped with an inherent capability to continuously adapt through online learning. This aspect remains in stark contrast to learning with error backpropagation through time (BPTT) applied to recurrent neural networks (RNNs), or recently to biologically-inspired spiking neural networks (SNNs). BPTT involves offline computation of the gradients due to the requirement to unroll the network through time. Online learning has recently regained the attention of the research community, focusing either on approaches that approximate BPTT or on biologically-plausible schemes applied to SNNs. Here we present an alternative perspective that is based on a clear separation of spatial and temporal gradient components. Combined with insights from biology, we derive from first principles a novel online learning algorithm for deep SNNs, called online spatio-temporal learning (OSTL). For shallow networks, OSTL is gradient-equivalent to BPTT enabling for the first time online training of SNNs with BPTT-equivalent gradients. In addition, the proposed formulation unveils a class of SNN architectures trainable online at low time complexity. Moreover, we extend OSTL to a generic form, applicable to a wide range of network architectures, including networks comprising long short-term memory (LSTM) and gated recurrent units (GRU). We demonstrate the operation of our algorithm on various tasks from language modelling to speech recognition and obtain results on par with the BPTT baselines. The proposed algorithm provides a framework for developing succinct and efficient online training approaches for SNNs and in general deep RNNs.


page 1

page 2

page 3

page 4


Long Short-Term Memory Spiking Networks and Their Applications

Recent advances in event-based neuromorphic systems have resulted in sig...

Online Spatio-Temporal Learning with Target Projection

Recurrent neural networks trained with the backpropagation through time ...

A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networks

We present a framework for compactly summarizing many recent results in ...

An Efficient EKF Based Algorithm For LSTM-Based Online Learning

We investigate online nonlinear regression with long short term memory (...

Efficient LSTM Training with Eligibility Traces

Training recurrent neural networks is predominantly achieved via backpro...

Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning

We systematically explored a spectrum of normalization algorithms relate...

A Microarchitecture Implementation Framework for Online Learning with Temporal Neural Networks

Temporal Neural Networks (TNNs) are spiking neural networks that use tim...

Please sign up or login with your details

Forgot password? Click here to reset