Restricted Boltzmann Machines as Models of Interacting Variables

03/29/2021
by   Nicola Bulso, et al.
0

We study the type of distributions that Restricted Boltzmann Machines (RBMs) with different activation functions can express by investigating the effect of the activation function of the hidden nodes on the marginal distribution they impose on observed binary nodes. We report an exact expression for these marginals in the form of a model of interacting binary variables with the explicit form of the interactions depending on the hidden node activation function. We study the properties of these interactions in detail and evaluate how the accuracy with which the RBM approximates distributions over binary variables depends on the hidden node activation function and on the number of hidden nodes. When the inferred RBM parameters are weak, an intuitive pattern is found for the expression of the interaction terms which reduces substantially the differences across activation functions. We show that the weak parameter approximation is a good approximation for different RBMs trained on the MNIST dataset. Interestingly, in these cases, the mapping reveals that the inferred models are essentially low order interaction models.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/23/2019

Learn-able parameter guided Activation Functions

In this paper, we explore the concept of adding learn-able slope and mea...
research
01/07/2019

On the effect of the activation function on the distribution of hidden nodes in a deep network

We analyze the joint probability distribution on the lengths of the vect...
research
12/13/2020

Using Restricted Boltzmann Machines to Model Molecular Geometries

Precise physical descriptions of molecules can be obtained by solving th...
research
06/24/2019

Variations on the Chebyshev-Lagrange Activation Function

We seek to improve the data efficiency of neural networks and present no...
research
08/30/2022

Transformers with Learnable Activation Functions

Activation functions can have a significant impact on reducing the topol...
research
01/27/2021

On the mapping between Hopfield networks and Restricted Boltzmann Machines

Hopfield networks (HNs) and Restricted Boltzmann Machines (RBMs) are two...
research
09/11/2018

Deep Asymmetric Networks with a Set of Node-wise Variant Activation Functions

This work presents deep asymmetric networks with a set of node-wise vari...

Please sign up or login with your details

Forgot password? Click here to reset