Regularized Dynamic Boltzmann Machine with Delay Pruning for Unsupervised Learning of Temporal Sequences

by   Sakyasingha Dasgupta, et al.

We introduce Delay Pruning, a simple yet powerful technique to regularize dynamic Boltzmann machines (DyBM). The recently introduced DyBM provides a particularly structured Boltzmann machine, as a generative model of a multi-dimensional time-series. This Boltzmann machine can have infinitely many layers of units but allows exact inference and learning based on its biologically motivated structure. DyBM uses the idea of conduction delays in the form of fixed length first-in first-out (FIFO) queues, with a neuron connected to another via this FIFO queue, and spikes from a pre-synaptic neuron travel along the queue to the post-synaptic neuron with a constant period of delay. Here, we present Delay Pruning as a mechanism to prune the lengths of the FIFO queues (making them zero) by setting some delay lengths to one with a fixed probability, and finally selecting the best performing model with fixed delays. The uniqueness of structure and a non-sampling based learning rule in DyBM, make the application of previously proposed regularization techniques like Dropout or DropConnect difficult, leading to poor generalization. First, we evaluate the performance of Delay Pruning to let DyBM learn a multidimensional temporal sequence generated by a Markov chain. Finally, we show the effectiveness of delay pruning in learning high dimensional sequences using the moving MNIST dataset, and compare it with Dropout and DropConnect methods.


page 1

page 5


Learning dynamic Boltzmann machines with spike-timing dependent plasticity

We propose a particularly structured Boltzmann machine, which we refer t...

Bio-plausible Unsupervised Delay Learning for Extracting Temporal Features in Spiking Neural Networks

The plasticity of the conduction delay between neurons plays a fundament...

Stochastic Synapses Enable Efficient Brain-Inspired Learning Machines

Recent studies have shown that synaptic unreliability is a robust and su...

Boltzmann machines for time-series

We review Boltzmann machines extended for time-series. These models ofte...

Efficient training of energy-based models via spin-glass control

We present an efficient method for unsupervised learning using Boltzmann...

Adaptive Sparse Structure Development with Pruning and Regeneration for Spiking Neural Networks

Spiking Neural Networks (SNNs) are more biologically plausible and compu...

Please sign up or login with your details

Forgot password? Click here to reset