Learning Restricted Boltzmann Machines via Influence Maximization
share
Graphical models are a rich language for describing high-dimensional distributions in terms of their dependence structure. While there are provable algorithms for learning graphical models in a variety of settings, there has been much less progress when there are latent variables. Here we study Restricted Boltzmann Machines (or RBMs), which are a popular model with wide-ranging applications in dimensionality reduction, collaborative filtering, topic modeling, feature extraction and deep learning. We give a simple greedy algorithm based on influence maximization to learn ferromagnetic RBMs with bounded degree. More precisely, we learn a description of the distribution on the observed variables as a Markov Random Field (or MRF), even though it exhibits complex higher- order interactions. Our analysis is based on tools from mathematical physics that were developed to show the concavity of magnetization. Moreover our results extend in a straightforward manner to ferromagnetic Ising models with latent variables. Conversely, we show that the distribution on the observed nodes of a general RBM can simulate any MRF which allows us to show new hardness results for improperly learning RBMs even with only a constant number of latent variables.
READ FULL TEXT