1 Introduction
Calculating marginal probabilities for a graphical model generally requires summing over exponentially many states, and is NPhard in general [1]. A variety of approximate methods have been used to circumvent this problem. One popular technique is belief propagation (BP), in particular the sumproduct rule, which is a messagepassing algorithm for performing inference on a graphical model [2]. Though exact and efficient on trees, it is merely an approximation when applied to graphical models with loops.
A natural question is whether one can compensate for the shortcomings of the approximation by setting the model parameters appropriately. In this paper, we prove that some sets of marginals simply cannot be achieved by belief propagation. For these cases we provide a new algorithm that can achieve much better results by using an ensemble of parameters rather than a single instance.
We are given a set of variables with a given probability distribution of some data. We would like to construct a model that reproduces certain of its marginal probabilities, in particular those over individual variables for nodes , and those over some relevant clusters of variables, for
. We will write the collection of all these marginals as a vector
.We assume a model distribution in the exponential family taking the form
(1) 
with normalization constant and energy function
(2) 
Here, indexes sets of interacting variables (factors in the factor graph [3]), and is a subset of variables whose interaction is characterized by a vector of sufficient statistics and corresponding natural parameters . We assume without loss of generality that each is irreducible, meaning that it cannot be written as a sum of any linearly independent functions that themselves do not depend on any for . We collect all these sufficient statistics and natural parameters in the vectors and .
Normally when learning a graphical model, one would fit its parameters so the marginal probabilities match the target. Here, however, we will not use exact inference to compute the marginals. Instead we will use approximate inference via loopy belief propagation to match the target.
2 Learning in Belief Propagation
2.1 Belief propagation
The sumproduct algorithm for belief propagation on a graphical model with energy function (2) uses the following equations [4]:
(3) 
where and are the neighbors of node or factor in the factor graph. Once these messages converge, the singlenode and factor beliefs are given by
(4) 
where the beliefs must each be normalized to one. For tree graphs, these beliefs exactly equal the marginals of the graphical model . For loopy graphs, the beliefs at fixed points are often good approximations of the marginals. While they are guaranteed to be locally consistent,
, they are not necessarily globally consistent: There may not exist a single joint distribution
of which the beliefs are the marginals [5]. This is why the resultant beliefs are called pseudomarginals, rather than simply marginals. We use a vector to refer to the set of both node and factor beliefs produced by belief propagation.2.2 Bethe free energy
Despite its limitations, BP is found empirically to work well in many circumstances. Some theoretical justification for loopy belief propagation emerged with proofs that its stable fixed points are local minima of the Bethe free energy [6, 7]
. Free energies are important quantities in machine learning because the KullbackLeibler divergence between the data and model distributions can be expressed in terms of free energies, so models can be optimized by minimizing free energies appropriately.
Given an energy function from (2), the Gibbs free energy of a distribution is
(5) 
where is the average energy of the distribution
(6) 
which depends on the marginals of , and is the entropy
(7) 
Minimizing the Gibbs free energy recovers the distribution for the graphical model (1).
The Bethe free energy is an approximation to the Gibbs free energy,
(8) 
in which the average energy is exact, but the true entropy is replaced by an approximation, the Bethe entropy , which is a sum over the factor and node entropies [6]:
(9) 
(10) 
The coefficients are the number of factors neighboring node , and compensate for the overcounting of singlenode marginals due to overlapping factor marginals. For treestructured graphical models, which factorize as , the Bethe entropy is exact, and hence so is the Bethe free energy. On loopy graphs, the Bethe entropy isn’t really even an entropy (e.g. it may be negative) because it neglects all statistical dependencies other than those present in the factor marginals. Nonetheless, the Bethe free energy is often close enough to the Gibbs free energy that its minima approximate the true marginals [8]. Since stable fixed points of BP are minima of the Bethe free energy [6, 7], this helped explain why belief propagation is often so successful.
To emphasize that the Bethe free energy directly depends only on the marginals and not the joint distribution, we will write where is a vector of pseudomarginals for all and all . Pseudomarginal space is the convex set [5] of all that satisfy the positivity and local consistency constraints,
(11) 
2.3 Pseudomoment matching
We now wish to correct for the deficiencies of belief propagation by identifying the parameters so that BP produces beliefs matching the true marginals of the target distribution . Since the fixed points of BP are stationary points of [6], one may simply try to find parameters that produce a stationary point in pseudomarginal space at , which is a necessary condition for BP to reach a fixed point there. Simply evaluate the gradient at , set it to zero, and solve for .
Note that in principle this gradient could be used to directly minimize the Bethe free energy, but is a complicated function of that usually cannot be minimized analytically [8]. In contrast, here we are using it to solve for the parameters needed to move beliefs to a target location. This is much easier, since the Bethe free energy is linear in
. This approach to learning parameters has been described as ‘pseudomoment matching’
[9, 10, 11].The element vector is an overcomplete representation of the pseudomarginals because it must obey the local consistency constraints (11). It is convenient to express the pseudomarginals in terms of a minimal set of parameters with the smaller dimensionality as and , using an affine transform
(12) 
where is an rectangular matrix. One example is the expectation parameters [5], giving the energy simply as . The gradient with respect to those minimal parameters is
(13) 
The Bethe entropy gradient is simplest in the overcomplete representation ,
(14) 
Setting the gradient (13) to zero, we have a simple linear equation for the parameters that tilt the Bethe free energy surface (Figure 1A) enough to place a stationary point at the desired marginals :
(15) 
2.4 Unbelievable marginals
It is well known that BP may converge on fixed points that cannot be realized as marginals of any joint distribution. In this section we show that the converse is also true: There are some distributions whose marginals cannot be realized as beliefs for any set of couplings. In these cases, existing methods for learning often yield poor results, sometimes even worse than performing no learning at all. This is surprising in view of claims to the contrary: [9, 5] state that belief propagation run after pseudomoment matching can always reach a fixed point that reproduces the target marginals. While BP does technically have such fixed points, they are not always stable and thus may not be reachable by running belief propagation.
Definition 1.
A set of marginals are ‘unbelievable’ if belief propagation cannot converge to them for any set of parameters.
For belief propagation to converge to the target — namely, the marginals — a zero gradient is not sufficient: The Bethe free energy must also be a local minimum [7].^{1}^{1}1Even this is not sufficient, but it is necessary. This requires a positivedefinite Hessian of (the ‘Bethe Hessian’ ) in the subspace of pseudomarginals that satisfies the local consistency constraints. Since the energy is linear in the pseudomarginals, the Hessian is given by the second derivative of the Bethe entropy,
(16) 
where projection by constrains the derivatives to the subspace spanned by the minimal parameters . If this Hessian is positive definite when evaluated at then the parameters given by (15) give a minimum at the target . If not, then the target cannot be a stable fixed point of loopy belief propagation. In Section 3, we calculate the Bethe Hessian explicitly for a binary model with pairwise interactions.
Theorem 1.
Unbelievable marginal probabilities exist.
Proof.
Proof by example. The simplest unbelievable example is a binary graphical model with pairwise interactions between four nodes, , and the energy . By symmetry and (1), marginals of this target are the same for all nodes and pairs: and . Substituting these marginals into the appropriate Bethe Hessian (22) gives a matrix that has a negative eigenvalue for all , or
. The associated eigenvector
has the same symmetry as the marginals, with singlenode components and pairwise components . Thus the Bethe free energy does not have a minimum at the marginals of these . Stable fixed points of BP occur only at local minima of the Bethe free energy [7], and so BP cannot reproduce the marginals for any parameters. Hence these marginals are unbelievable. ∎Not only do unbelievable marginals exist, but they are actually quite common, as we will see in Section 3. Graphical models with multinomial or gaussian variables and at least two loops always have some pseudomarginals for which the Hessian is not positive definite [12]. On the other hand, all marginals with sufficiently small correlations are believable because they are guaranteed to have a positivedefinite Bethe Hessian [12]. Stronger conditions have not yet been described.
2.5 Bethe wakesleep algorithm
When pseudomoment matching fails to reproduce unbelievable marginals, an alternative is to use a gradient descent procedure for learning, analagous to the wakesleep algorithm used to train Boltzmann machines
[13]. The original rule can be derived as gradient descent of the KullbackLeibler divergence between the target and the graphical model (1),(17) 
where is the Gibbs free energy (5) using the energy function (2). Here we use a new cost function, the ‘Bethe divergence’ , by replacing these free energies by Bethe free energies [14] evaluated at the true marginals and at the beliefs obtained from BP fixed points,
(18) 
We use gradient descent to optimize this cost, with gradient
(19) 
The data’s free energy does not depend on the beliefs, so , and fixed points of belief propagation are stationary points of the Bethe free energy, so . Consequently . Furthermore, the entropy terms of the free energies do not depend explicitly on , so
(20) 
where are the expectations of the sufficient statistics under the pseudomarginals . This gradient forms the basis of a simple learning algorithm. At each step in learning, belief propagation is run, obtaining beliefs for the current parameters . The parameters are then changed in the opposite direction of the gradient,
(21) 
where is a learning rate. This generally increases the Bethe free energy for the beliefs while decreasing that of the data, hopefully allowing BP to draw closer to the data marginals. We call this learning rule the Bethe wakesleep algorithm.
Within this algorithm, there is still the freedom of how to choose initial messages for BP at each learning iteration. The result depends on these initial conditions because BP can have several stable fixed points. One might reinitialize the messages to a fixed starting point for each run of BP, choose random initial messages for each run, or restart the messages where they stopped on the previous learning step. In our experiments we use the first approach, initializing to constant messages at the beginning of each BP run.
The Bethe wakesleep learning rule sometimes places a minimum of at the true data distribution, such that belief propagation can give the true marginals as one of its (possibly multiple) fixed points. However, for the reasons provided above, this cannot occur where the Bethe Hessian is not positive definite.
2.6 Ensemble belief propagation
When the Bethe wakesleep algorithm attempts to learn unbelievable marginals, the parameters and beliefs do not reach a fixed point but instead continue to vary over time (Figure 2A,B). Still, if learning reaches equilibrium, then the temporal average of beliefs is equal to the unbelievable marginals.
Theorem 2.
If the Bethe wakesleep algorithm reaches equilibrium, then unbelievable marginals are matched by the belief propagation fixed points averaged over the equilibrium ensemble of parameters.
Proof.
After learning has equilibrated, fixed points of belief propagation occur with just the right frequency so that they can be averaged together to reproduce the target distribution exactly (Figure 2C). Note that none of the individual fixed points may be close to the true marginals. We call this inference algorithm ensemble belief propagation (eBP).
Ensemble BP produces perfect marginals by exploiting a constant, small amplitude learning, and thus assumes that the correct marginals are perpetually available. Yet it also works well when learning is turned off, if parameters are drawn randomly from a gaussian distribution with mean and covariance matched to the equilibrium distribution,
. In the simulations below (Figures 2C–D, 3B–C), was always lowrank, and only one or two principle components were needed for good performance. The gaussian ensemble is not quite as accurate as continued learning (Figure 3B,C), but the performance is still markedly better than any of the available fixed points.If the target is not within a convex hull of believable pseudomarginals, then learning cannot reach equilibrium: Eventually BP gets as close as it can but there remains a consistent difference , so must increase without bound. Though possible in principle, we did not observe this effect in any of our experiments. There may also be no equilibrium if belief propagation at each learning iteration fails to converge.
3 Experiments
The experiments in this section concentrate on the Ising model: binary variables, , with factors comprising individual variables and pairs , . The energy function is . Then the sufficient statistics are the various first and second moments, and , and the natural parameters are , . We use this model both for the target distributions and the model.
We parameterize pseudomarginals as where and [8]. The remaining probabilities are linear functions of these values. Positivity constraints and local consistency constraints then appear as and . If all the interactions are finite, then the inequality constraints are not active [15]. In this parameterization, the elements of the Bethe Hessian (16) are
(22a)  
(22b)  
(22c) 
Figure 3A shows the fraction of marginals that are unbelievable for 8node, fullyconnected Ising models with random coupling parameters and . For , most marginals cannot be reproduced by belief propagation with any parameters, because the Bethe Hessian (22) has a negative eigenvalue.
We generated 500 Ising model targets using , selected the unbelievable ones, and evaluated the performance of BP and ensemble BP for various methods of choosing parameters . Each run of BP used exponential temporal message damping of 5 time steps [16], with . Fixed points were declared when messages changed by less than on a single time step. We evaluated BP performance for the actual parameters that generated the target (1), pseudomoment matching (15), and at bestmatching beliefs obtained at any time during Bethe wakesleep learning. We also measured eBP performance for two parameter ensembles: the last 100 iterations of Bethe wakesleep learning, and parameters sampled from a gaussian with the same mean and covariance as that ensemble.
Belief propagation gave a poor approximation of the target marginals, as expected for a model with many strong loops. Even with learning, BP could never get the correct marginals, which was guaranteed by selection of unbelievable targets. Yet ensemble belief propagation gave excellent results. Using the exact parameter ensemble gave orders of magnitude improvement, limited by the number of beliefs being averaged. The gaussian parameter ensemble also did much better than even the best results of BP.
4 Discussion
Other studies have also made use of the Bethe Hessian to draw conclusions about belief propagation. For instance, the Hessian reveals that the Ising model’s paramagnetic state becomes unstable in BP for large enough couplings [17]. For another example, when the Hessian is positive definite throughout pseudomarginal space, then the Bethe free energy is convex and thus BP has a unique fixed point [18]. Yet the stronger interpretation appears to be underappreciated: When the Hessian is not positive definite for some pseudomarginals, then BP can never have a fixed point there, for any parameters.
One might hope that by adjusting the parameters of belief propagation in some systematic way, , one could fix the approximation and so perform exact inference. In this paper we proved that this is a futile hope, because belief propagation simply can never converge to certain marginals. However, we also provided an algorithm that does work: Ensemble belief propagation uses BP on several different parameters with different fixed points and averages the results. This approach preserves the locality and scalability which make BP so popular, but corrects for some of its defects at the cost of running the algorithm a few times. Additionally, it raises the possibility that a systematic compensation for the flaws of BP might exist, but only as a mapping from individual parameters to an ensemble of parameters that could be used in eBP.
An especially clear application of eBP is to discriminative models like Conditional Random Fields [19]. These models are trained so that known inputs produce known inferences, and then generalize to draw novel inferences from novel inputs. When belief propagation is used during learning, then the model will fail even on known training examples if they happen to be unbelievable. Overall performance will suffer. Ensemble BP can remedy those training failures and thus allow better performance and more reliable generalization.
This paper addressed learning in fullyobserved models only, where marginals for all variables were available during training. Yet unbelievable marginals exist for models with hidden variables as well. Ensemble BP should work as in the fullyobserved case, but training will require inference over the hidden variables during both wake and sleep phases.
One important inference engine is the brain. When inference is hard, neural computations may resort to approximations, perhaps including belief propagation [20, 21, 22, 23, 24]. It would be undesirable for neural circuits to have big blind spots, i.e. reasonable inferences it cannot draw, yet that is precisely what occurs in BP. By averaging over models with eBP, this blind spot can be eliminated. In the brain, synaptic weights fluctuate due to a variety of mechanisms. Perhaps such fluctuations allow averaging over models and thereby reach conclusions unattainable by a deterministic mechanism.
Acknowledgments
The authors thank Greg Wayne for helpful conversations.
References
 [1] Cooper G (1990) The computational complexity of probabilistic inference using bayesian belief networks. Artificial intelligence 42: 393–405.
 [2] Pearl J (1988) Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan Kaufmann Publishers, San Mateo CA.
 [3] Kschischang F, Frey B, Loeliger H (2001) Factor graphs and the sumproduct algorithm. IEEE Transactions on Information Theory 47: 498–519.

[4]
Bishop C (2006) Pattern recognition and machine learning.
Springer New York.  [5] Wainwright M, Jordan M (2008) Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning 1: 1–305.
 [6] Yedidia JS, Freeman WT, Weiss Y (2000) Generalized belief propagation. In: IN NIPS 13. MIT Press, pp. 689–695.
 [7] Heskes T (2003) Stable fixed points of loopy belief propagation are minima of the Bethe free energy. Advances in Neural Information Processing Systems 15: 343–350.
 [8] Welling M, Teh Y (2001) Belief optimization for binary networks: A stable alternative to loopy belief propagation. In: Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence. Morgan Kaufmann Publishers Inc., pp. 554–561.

[9]
Wainwright MJ, Jaakkola TS, Willsky AS (2003) Treereweighted belief propagation algorithms and approximate ML estimation by pseudomoment matching.
In: AISTATS.  [10] Welling M, Teh Y (2003) Approximate inference in Boltzmann machines. Artificial Intelligence 143: 19–50.
 [11] Parise S, Welling M (2005) Learning in markov random fields: An empirical study. In: Joint Statistical Meeting. volume 4.
 [12] Watanabe Y, Fukumizu K (2011) Loopy belief propagation, Bethe free energy and graph zeta function. arXiv cs.AI.
 [13] Hinton G, Sejnowski T (1983) Analyzing cooperative computation. Proceedings of the Fifth Annual Cognitive Science Society, Rochester NY .
 [14] Welling M, Sutton C (2005) Learning in markov random fields with contrastive free energies. In: Cowell RG, Ghahramani Z, editors, In Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics. Society for Artificial Intelligence and Statistics, pp. 397404.
 [15] Yedidia J, Freeman W, Weiss Y (2005) Constructing freeenergy approximations and generalized belief propagation algorithms. IEEE Transactions on Information Theory 51: 2282–2312.
 [16] Mooij J, Kappen H (2005) On the properties of the Bethe approximation and loopy belief propagation on binary networks. Journal of Statistical Mechanics: Theory and Experiment : P11012.
 [17] Mooij J, Kappen H (2005) Validity estimates for loopy belief propagation on binary realworld networks. In: Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, pp. 945–952.
 [18] Heskes T (2004) On the uniqueness of loopy belief propagation fixed points. Neural Computation 16: 2379–2413.
 [19] Lafferty J, McCallum A, Pereira F (2001) Conditional random fields: Probabilistic models for segmenting and labeling sequence data. Proceedings of the 18th International Conference on Machine Learning : 282–289.
 [20] Litvak S, Ullman S (2009) Cortical circuitry implementing graphical models. Neural Computation 21: 3010–3056.

[21]
Steimer A, Maass W, Douglas R (2009) Belief propagation in networks of spiking neurons.
Neural Computation 21: 2502–2523.  [22] Ott T, Stoop R (2007) The neurodynamics of belief propagation on binary markov random fields. In: Schölkopf B, Platt J, Hoffman T, editors, Advances in Neural Information Processing Systems 19, Cambridge, MA: MIT Press. pp. 1057–1064.
 [23] Shon A, Rao R (2005) Implementing belief propagation in neural circuits. Neurocomputing 65–66: 393–399.
 [24] George D, Hawkins J (2009) Towards a mathematical theory of cortical microcircuits. PLoS computational biology 5: 1–26.
Comments
There are no comments yet.