A topological insight into restricted Boltzmann machines

Restricted Boltzmann Machines (RBMs) and models derived from them have been successfully used as basic building blocks in deep artificial neural networks for automatic features extraction, unsupervised weights initialization, but also as density estimators. Thus, their generative and discriminative capabilities, but also their computational time are instrumental to a wide range of applications. Our main contribution is to look at RBMs from a topological perspective, bringing insights from network science. Firstly, here we show that RBMs and Gaussian RBMs (GRBMs) are bipartite graphs which naturally have a small-world topology. Secondly, we demonstrate both on synthetic and real-world datasets that by constraining RBMs and GRBMs to a scale-free topology (while still considering local neighborhoods and data distribution), we reduce the number of weights that need to be computed by a few orders of magnitude, at virtually no loss in generative performance. Thirdly, we show that, for a fixed number of weights, our proposed sparse models (which by design have a higher number of hidden neurons) achieve better generative capabilities than standard fully connected RBMs and GRBMs (which by design have a smaller number of hidden neurons), at no additional computational costs.

READ FULL TEXT
research
07/15/2017

Evolutionary Training of Sparse Artificial Neural Networks: A Network Science Perspective

Through the success of deep learning, Artificial Neural Networks (ANNs) ...
research
01/21/2019

On Compression of Unsupervised Neural Nets by Pruning Weak Connections

Unsupervised neural nets such as Restricted Boltzmann Machines(RBMs) and...
research
07/17/2015

Massively Deep Artificial Neural Networks for Handwritten Digit Recognition

Greedy Restrictive Boltzmann Machines yield an fairly low 0.72 the famou...
research
07/10/2018

An Adaptive Learning Method of Restricted Boltzmann Machine by Neuron Generation and Annihilation Algorithm

Restricted Boltzmann Machine (RBM) is a generative stochastic energy-bas...
research
05/11/2023

Investigating the generative dynamics of energy-based neural networks

Generative neural networks can produce data samples according to the sta...
research
12/27/2019

Emergence of Network Motifs in Deep Neural Networks

Network science can offer fundamental insights into the structural and f...

Please sign up or login with your details

Forgot password? Click here to reset