Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses

12/06/2016
by   Haiping Huang, et al.
0

Revealing hidden features in unlabeled data is called unsupervised feature learning, which plays an important role in pretraining a deep neural network. Here we provide a statistical mechanics analysis of the unsupervised learning in a restricted Boltzmann machine with binary synapses. A message passing equation to infer the hidden feature is derived, and furthermore, variants of this equation are analyzed. A statistical analysis by replica theory describes the thermodynamic properties of the model. Our analysis confirms an entropy crisis preceding the non-convergence of the message passing equation, suggesting a discontinuous phase transition as a key characteristic of the restricted Boltzmann machine. Continuous phase transition is also confirmed depending on the embedded feature strength in the data. The mean-field result under the replica symmetric assumption agrees with that obtained by running message passing algorithms on single instances of finite sizes. Interestingly, in an approximate Hopfield model, the entropy crisis is absent, and a continuous phase transition is observed instead. We also develop an iterative equation to infer the hyper-parameter (temperature) hidden in the data, which in physics corresponds to iteratively imposing Nishimori condition. Our study provides insights towards understanding the thermodynamic properties of the restricted Boltzmann machine learning, and moreover important theoretical basis to build simplified deep networks.

READ FULL TEXT
research
02/01/2015

Advanced Mean Field Theory of Restricted Boltzmann Machine

Learning in restricted Boltzmann machine is typically hard due to the co...
research
02/23/2015

Approximate Message Passing with Restricted Boltzmann Machine Priors

Approximate Message Passing (AMP) has been shown to be an excellent stat...
research
11/11/2019

How data, synapses and neurons interact with each other: a variational principle marrying gradient ascent and message passing

Unsupervised learning requiring only raw data is not only a fundamental ...
research
05/04/2020

A Dynamical Mean-Field Theory for Learning in Restricted Boltzmann Machines

We define a message-passing algorithm for computing magnetizations in Re...
research
04/30/2019

Minimal model of permutation symmetry in unsupervised learning

Permutation of any two hidden units yields invariant properties in typic...
research
01/03/2020

A Probability Density Theory for Spin-Glass Systems

Spin-glass systems are universal models for representing many-body pheno...
research
11/08/2017

Approximate message passing for nonconvex sparse regularization with stability and asymptotic analysis

We analyze linear regression problem with a nonconvex regularization cal...

Please sign up or login with your details

Forgot password? Click here to reset