Deep learning via message passing algorithms based on belief propagation

10/27/2021
by   Carlo Lucibello, et al.
0

Message-passing algorithms based on the Belief Propagation (BP) equations constitute a well-known distributed computational scheme. It is exact on tree-like graphical models and has also proven to be effective in many problems defined on graphs with loops (from inference to optimization, from signal processing to clustering). The BP-based scheme is fundamentally different from stochastic gradient descent (SGD), on which the current success of deep networks is based. In this paper, we present and adapt to mini-batch training on GPUs a family of BP-based message-passing algorithms with a reinforcement field that biases distributions towards locally entropic solutions. These algorithms are capable of training multi-layer neural networks with discrete weights and activations with performance comparable to SGD-inspired heuristics (BinaryNet) and are naturally well-adapted to continual learning. Furthermore, using these algorithms to estimate the marginals of the weights allows us to make approximate Bayesian predictions that have higher accuracy than point-wise solutions.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/05/2022

Nonlocal optimization of binary neural networks

We explore training Binary Neural Networks (BNNs) as a discrete variable...
research
01/15/2014

Join-Graph Propagation Algorithms

The paper investigates parameterized approximate message-passing schemes...
research
09/24/2022

Deep Attentive Belief Propagation: Integrating Reasoning and Learning for Solving Constraint Optimization Problems

Belief Propagation (BP) is an important message-passing algorithm for va...
research
09/24/2019

Message Scheduling for Performant, Many-Core Belief Propagation

Belief Propagation (BP) is a message-passing algorithm for approximate i...
research
02/07/2012

Message-Passing Algorithms for Channel Estimation and Decoding Using Approximate Inference

We design iterative receiver schemes for a generic wireless communicatio...
research
09/02/2013

Sigma Point Belief Propagation

The sigma point (SP) filter, also known as unscented Kalman filter, is a...
research
01/26/2014

Perturbed Message Passing for Constraint Satisfaction Problems

We introduce an efficient message passing scheme for solving Constraint ...

Please sign up or login with your details

Forgot password? Click here to reset