Belief-propagation algorithm and the Ising model on networks with arbitrary distributions of motifs

06/24/2011
by   S. Yoon, et al.
0

We generalize the belief-propagation algorithm to sparse random networks with arbitrary distributions of motifs (triangles, loops, etc.). Each vertex in these networks belongs to a given set of motifs (generalization of the configuration model). These networks can be treated as sparse uncorrelated hypergraphs in which hyperedges represent motifs. Here a hypergraph is a generalization of a graph, where a hyperedge can connect any number of vertices. These uncorrelated hypergraphs are tree-like (hypertrees), which crucially simplify the problem and allow us to apply the belief-propagation algorithm to these loopy networks with arbitrary motifs. As natural examples, we consider motifs in the form of finite loops and cliques. We apply the belief-propagation algorithm to the ferromagnetic Ising model on the resulting random networks. We obtain an exact solution of this model on networks with finite loops or cliques as motifs. We find an exact critical temperature of the ferromagnetic phase transition and demonstrate that with increasing the clustering coefficient and the loop size, the critical temperature increases compared to ordinary tree-like complex networks. Our solution also gives the birth point of the giant connected component in these loopy networks.

READ FULL TEXT VIEW PDF

page 1

page 2

page 3

page 4

04/12/2017

Beliefs in Markov Trees - From Local Computations to Local Valuation

This paper is devoted to expressiveness of hypergraphs for which uncerta...
04/20/2011

Palette-colouring: a belief-propagation approach

We consider a variation of the prototype combinatorial-optimisation prob...
10/24/2018

Loop corrections in spin models through density consistency

Computing marginal distributions of discrete or semi-discrete Markov Ran...
06/20/2018

Efficient inference in stochastic block models with vertex labels

We study the stochastic block model with two communities where vertices ...
03/08/2022

Online Target Localization using Adaptive Belief Propagation in the HMM Framework

This paper proposes a novel adaptive sample space-based Viterbi algorith...
07/31/2013

A Time and Space Efficient Junction Tree Architecture

The junction tree algorithm is a way of computing marginals of boolean m...
10/16/2012

A Cluster-Cumulant Expansion at the Fixed Points of Belief Propagation

We introduce a new cluster-cumulant expansion (CCE) based on the fixed p...

I Introduction

The belief-propagation algorithm is an effective numerical method for solving inference problems on sparse graphs. It was originally proposed by J. Pearl Pearl:p88

for tree-like graphs. The belief-propagation algorithm was applied to study diverse systems in computer science, physics, and biology. Among its numerous applications are computer vision problems, decoding of high performance turbo codes and many others, see

Frey:98 ; McEliece:mmc98 . Empirically, it was found that this algorithm works surprisingly good even for graphs with loops. J. S. Yedidia et al. Yedidia:yfw01 found that the belief-propagation algorithm actually coincides with the minimization of the Bethe free energy. This discovery renewed interest in the Bethe-Peierls approximation and related methods Pretti:pp03 ; Mooij:mk05 ; Hartmann:hw05 . In statistical physics the belief-propagation algorithm is equivalent to the so-called cavity method proposed by Mézard, Parisi, and Virasoro mp2003 . The belief-propagation algorithm is applicable to models with both discrete and continuous variables mooij04 ; ohkubo05 ; Dorogovtsev:dgm08 . Recently the belief-propagation algorithm was applied to diverse problems in random networks: spread of disease karrer10 , the calculation of the size of the giant component sk10 , counting large loops in directed random uncorrelated networks bianconi08 , the graph bi-partitioning problem sulc10 , analysis of the contributions of individual nodes and groups of nodes to the network structure ras10

, and networks of spiking neurons

smd09 .

These investigations showed that the belief-propagation algorithm is exact for a model on a graph with locally tree-like structure in the infinite size limit. However, only an approximate solution was obtained by use of this method for networks with short loops. Real technological, social and biological networks have numerous short and large loops and other complex subgraphs or motifs which lead to essentially non-tree-like neighborhoods ab01a ; dm01c ; Dorogovtsev:2010 ; Newman:n03a ; Milo2002 ; Milo2004 ; Sporn2004 ; Alon2007 . That is why it is important to develop a method which takes into account finite loops and more complex motifs. Different ways were proposed recently to compute loop corrections to the Bethe approximation by use of the belief-propagation algorithm mr05 ; ps05 . These methods, however, are exact only if a graph contains a single loop. Loop expansions were proposed in Refs. Chertkov:cc06a ; Chertkov:cc06b . However, they were not applied to complex network yet.

Recently, Newman and Miller newman:n09 ; m09 ; kn10b independently introduced a model of random graphs with arbitrary distributions of subgraphs or motifs. In this natural generalization of the configuration model, each vertex belongs to a given set of motifs [e.g., vertex is a member of motifs of type , motifs of type , and so on], and apart from this constraint, the network is uniformly random. In the original configuration model, the sequence of vertex degrees is fixed, where . In this generalization, the sequence of generalized vertex degrees is given. For example, motifs can be triangles, loops, chains, cliques (fully connected subgraphs), single edges that do not enter other motifs, and, in general, arbitrary finite clusters. The resulting networks can be treated as uncorrelated hypergraphs in which hyperedges represent motifs. In graph theory, a hypergraph is a generalization of a graph, where a hyperedge can connect any number of vertices Berge73 . Because of the complex motifs, the large sparse networks under consideration can have loops, clustering, and correlations, but the underlying hypergraphs are locally tree-like (hypertrees) and uncorrelated similarly to the original sparse configuration model. Our approach is based on the hypertree-like structure of these highly structured sparse networks. To demonstrate our approach, we apply the generalized belief-propagation algorithm to the ferromagnetic Ising model on the sparse networks in which motifs are finite loops or cliques. We obtain an exact solution of the ferromagnetic Ising model and the birth point of the giant connected component in this kind of highly structured networks. Note that the Ising model on these networks is a more complex problem than the percolation problem because we must account for spin interactions between spins both inside and between motifs. We find an exact critical temperature of the ferromagnetic phase transition and demonstrate that finite loops increase the critical temperature in comparison to ordinary tree-like networks.

Ii Ensemble of random networks with motifs

Let us introduce a statistical ensemble of random networks with a given distribution of motifs in which each vertex belongs to a given set of motifs and apart of this constraint, the networks are uniformly random. These networks can be treated as uncorrelated hypergraphs, in which motifs play a role of hyperedges, so the number of hyperdegrees are equal to the number of specific motifs attached to a vertex. In principal, one can choose any subgraph with an arbitrary number of vertices as a motif, see Fig. 1. In the present paper, for simplicity, we only consider simple motifs such as single edges, finite loops, and cliques.

Let us first note how one can describe a statistical ensemble of random networks with the simplest motifs, namely simple edges (see, for example, Refs. Dorogovtsev:dgm08 ; Bianconi09

and references therein). We define the probability

that an edge between vertices and is present () or absent (),

(1)

where are entries of the symmetrical adjacency matrix, is the mean number of edges attached to a vertex. It is well known that the probability of the realization of the Erdős-Rényi graph with a given adjacency matrix , is the product

(2)

The degree distribution is the Poisson distribution. In the configuration model, the probability of the realization of a given graph with a given sequence of degrees,

, , ,…, , is

(3)

The delta-function fixes the number of edges attached to vertex . is a normalization constant. The distribution function of degrees is determined by the sequence ,

(4)

The second simplest motif, the triangle, plays the role of a hyperedge that interconnects a triple of vertices. Let us introduce the probability that a hyperedge (triangle) among vertices , , and is present or absent, i.e., or , respectively:

(5)

where

(6)

is the probability that vertices and form a triangle. is the mean number of triangles attached to a randomly chosen vertex. are entries of the adjacency matrix of the hypergraph. This matrix is symmetrical with respect to permutations of the indices and ,   .

For example, one can introduce the ensemble of the Erdős-Rényi hypergraphs. Given that the matrix elements are independent and uncorrelated random parameters, the probability of realization of a graph with a given adjacency matrix is the product of probabilities over different triples of vertices:

(7)

This function describes the ensemble of the Erdős-Rényi hypergraphs with the Poisson degree distributions of the number of triangles attached to vertices,

(8)

In the configuration model, the probability of the realization of a given graph with a sequence of the number of triangles, , , ,…,, attached to vertices is defined by an equation,

(9)

Here is given by Eq. (5). The delta-function fixes the number of triangles attached to vertex . is a normalization constant. The distribution function of triangles is

(10)
Figure 1: (Color online) Different motifs can be attached to vertex in a hypergraph: an ordinary edge , a triangle, a square, and a pentagon. There are also non-symmetric motifs consisting of four vertices and a clique of size 4. A motif can have internal vertices (open circle) that have edges only inside this motif. Motifs form a locally tree-like hypergraph where they play a role of hyperedges. Arrows represent incoming messages that arrive at vertex from the motifs attached to this vertex.

One can further generalize equations Eqs. (5) and (6) and introduce the probability that vertices and form a hyperedge of size 4 (a clique or loop of size 4). In the case of loops, one can arrange these vertices in order of increasing vertex index. Then, for a given sequence of squares or cliques, , attached to vertices , one can introduce the probability of the realization of a given graph with a given sequence similar to Eq. (9), and so on. The network ensemble of the configuration model with a given sequences of edges , triangles , squares and other motifs is described by a product of the corresponding probabilities,

(11)

The average of a quantity over the network ensemble is

(12)

where we integrate over all possible edges and hyperedges.

One can prove that the probability that different motifs have a common edge, i.e., they are overlapping, tends to zero in the infinite size limit . For example, the total number of edges that overlap with triangles equals

(13)

This number of double connections is finite in the limit . Because the total number of edges is of the order of , the fraction of double connections tends to zero in the thermodynamic limit. Thus, one can neglect overlapping among different motifs. Using the standard methods in complex network theory Bianconi:bc03 ; Bianconi:bm05a ; Newman:n03b , one can show that in the configuration model the number of finite loops formed by hyperedges is finite in the thermodynamic limit. Therefore, sparse random uncorrelated hypergraphs have a hypertree-like structure.

Uncorrelated random hypergraphs are described by a distribution function which is the probability that a randomly chosen vertex has edges, triangles, squares, and other motifs attached to this vertex. In the case of uncorrelated motifs, we have

(14)

Here, is the distribution function of loops of size ,

(15)

In this kind of network, the number of nearest neighboring vertices of vertex is equal to

(16)

Therefore, the mean degree is

(17)

The local clustering coefficient of node with degree , Eq. (16), is determined by the number of triangles as follows,

(18)

Therefore, is maximum if a network consists only of triangles. In this case, we have and therefore

(19)

On the other hand, if a network only consists of -cliques with a given (fully connected subgraphs of size ), then the local clustering coefficient of vertex with attached cliques equals

(20)

where is degree of vertex . If other motifs (edges, squares, and larger finite loops) are present in the network, then according to Eqs. (18) and (20) the local clustering coefficient is smaller than . In Refs. Serrano2006a ; Serrano2006b ; Serrano2006c , it was shown that properties of networks with ”weak clustering”, , may differ from properties of networks with ”strong clustering” when decreases slower than . The latter networks are beyond the scope of the present article.

Iii Belief-propagation algorithm

Let us consider the Ising model with pairwise interactions between nearest neighboring spins on a sparse random network with arbitrary distributions of motifs. The Hamiltonian of the model is

(21)

Here we sum the energies of spin clusters corresponding to motifs in the network. The second sum corresponds to simple edges. The third sum corresponds to triangles. The forth sum corresponds to motifs of size 4, and so on. is a local magnetic field at vertex . The coupling determines the energy of pairwise interaction between spins and . In general, one can introduce multi-spin interactions between spins in hyperedges, for example, , , and so on. However, this kind of interaction is beyond the scope of our article. Generally, the local fields and the couplings can be random quantities.

In order to solve this model, we generalize the belief-propagation algorithm. At first, we note the belief-propagation algorithm in application for the Ising model on tree-like uncorrelated complex networks without short loops (for more details, see Ref. Dorogovtsev:dgm08 ). In this case, there are only edges determined by the adjacency matrix and the clustering coefficient is zero in the thermodynamic limit. Consider spin . A nearest neighboring spin sends a so-called message to spin that we denote as . This message is normalized,

(22)

Within the belief-propagation algorithm, the probability that spin is in a state is determined by the normalized product of incoming messages to spin and the probabilistic factor due to a local field :

(23)

where is a normalization constant and

is the reciprocal temperature. Now we can calculate the mean moment of spin

,

(24)

A message can be written in a general form,

(25)

Using this representation, we rewrite Eq. (24) in a physically clear form,

(26)

This equation shows that a parameter plays the role of an effective field produced by spin at site . Messages obey a self-consistent equation that is called update rule,

(27)

where the index numerates nearest neighbors of vertex and is a normalization constant. The diagram representation of this equation is shown in Fig. 2 . The probabilistic factor takes account of the interaction energy of spins and and a local field ,

(28)

Thus, a message from to is determined by the coupling between these spins and messages received by from its neighbors except . Multiplying Eq. (27) by and summing over , we obtain a self-consistent equation for the effective fields ,

(29)

In the thermodynamic limit, this equation is exact for a tree-like graph. The Bethe-Peierls approach and the Baxter recurrence method give exactly the same result Dorogovtsev:dgm08 ; bbook82 . In numerical calculations, Eqs. (27) and (29) may be solved by use of iterations. In a general case, an analytical solution of these equations is unknown. In the case of all-to-all interactions with random couplings (the Sherrington-Kirkpatrick model), this set of equations is reduced to well-known TAP equations Thouless:tap77 that are exact in the thermodynamic limit. These equations also give an exact solution of the ferromagnetic Ising model with uniform couplings on a random uncorrelated graph with zero clustering coefficient dgm02 ; lvvz02 ; Dorogovtsev:dgm08 .

Figure 2: Diagram representation of the belief-propagation update rules for messages from a motif to a destination vertex . Arrows represent incoming messages. [(a), (b) and (c)] Update rules for an edge, a triangle, and a square. Mathematical expressions of these diagrams are given by Eqs. (27) and (33).

We now generalize the belief-propagation algorithm to the case of networks with given distributions of motifs described in Sec. II. In a network that consists only of edges, a message goes along an edge from the spin at one edge end to the spin at the other edge end. For the networks with motifs, it is natural to introduce a message that is sent by a motif attached to a vertex. This message goes to a spin to which this motif (hyperedge) is attached. Different motifs may be attached to a vertex, see Fig. 1. Let denote a cluster of size attached to vertex that we will call the destination vertex. Vertices together with the destination vertex form this motif. We introduce a message to spin from a motif . This message is normalized and can be written as follows,

(30)

As above, the probability that spin is in a state is determined by the normalized product of incoming messages from motifs attached to spin and the probabilistic factor ,

(31)

Using Eq. (24), we find that the mean moment is determined by effective fields acting on spin from motifs , , attached to , see Fig. 1,

(32)

Here the sum is taken over motifs attached to .

Let us find an update rule for a message from a given motif to vertex . We introduce the following update rule:

(33)

This rule shows that the message is equal to the product of incoming messages from motifs attached to all spins in the motif except spin and the motif itself (see Fig. 2. In Eq. (33) we average over all spin configurations of the spins , and in the motif. is a normalization constant. is an energy of the interaction between spins in the motif . This update rule also takes account of local fields acting on all spins except the destination spin . We note that Eq. (33) is valid for arbitrary motifs even with a complex internal structure. The only assumption is that a sparse network under consideration, in terms of hypergraphs, has a hypertree-like structure. Messages can be found numerically by use of iterations of Eq. (33) that start from an initial distribution of the messages.

For the sake of simplicity, let motifs be finite loops of size . Then

(34)

where . If motifs are -cliques, then the energy takes account of interactions between all spins in these cliques,

(35)

Multiplying Eq. (33) by and summing over all spin configurations, we obtain an equation for the effective field ,

(36)

The function on the right hand side is

(37)

This function has a meaning of the mean moment of spin in the cluster with an energy

(38)

This energy takes into account both the interaction between spins in this motif and total effective fields acting on these spins. In turn, the field acting on is the sum of a local field and effective fields produced by incoming messages from motifs attached to vertex except the motif , see Fig. 3,

(39)

is the partition function of the cluster ,

(40)

Thus, the function is a function of the total fields acting on all spins in the motif except spin . For a given network with motifs, it is necessary to solve Eq. (36) with respect to the effective fields . One then can calculate the mean magnetic moments from Eq. (32). Note that the only condition we used to derive the equations above was hypertree-like structure of the networks. The absence of correlations in these hypertree-like networks was not needed. Equations (33) and  (36) are our main result that actually generalizes the Bethe-Peierls approach and the Baxter recurrence method.

Let us study the ferromagnetic Ising model with uniform couplings, , at zero magnetic field, i.e., , on a uncorrelated hypergraph which consists of edges and finite loops. In a random network, effective fields

are also random variable and fluctuate from vertex to vertex. For a given

, we introduce the distribution function of fields ,

(41)

Here, we sum over vertices and attached motifs of size . is the normalization constant. We assume that in the thermodynamic limit, , the average over vertices in the network is equivalent to the average over the statistical network ensemble, Eq. (12). Using Eq. (36), we obtain an equation for the distribution function ,

(42)

Here is the function defined by Eq. (37). is a total field, Eq. (39), acting on vertex in the motif , see Fig. 3. The integration is over fields with the distribution function . Using Eq. (39), we can find this function,

Here is given by Eq. (14). This is the probability that a destination vertex has edges, triangles, and so on. In turn, incoming messages to the destination spin from attached loops of size are numerated by the index , . Only for incoming messages from loops of size do we have , because we should not account for the motif . The integration is over incoming messages to all vertices in the motif (hyperedge) except the destination vertex , see Fig. 3. Note that we have an additional factor , because there is the probability in the configuration model that if we arrive at a destination vertex along a hyperedge , then this vertex has outgoing hyperedges . Equations  (42) and  (LABEL:phi) represent a set of self-consistent equations for the functions , . Using Eq. (32), one then can find a distribution function of spontaneous magnetic moments in the network. Equations  (42) and  (LABEL:phi) are also valid for networks with cliques.

Figure 3: (a) A loop (motif ) of size is attached to destination vertex . Wave arrows represent total effective fields acting on spins with index in this loop. (b) In turn, the total effective field is a sum of effective fields produced by all motifs attached to vertex except the motif . There are edges, triangles, and so on. However, the number of loops of size equals , because we should not account the motif that is common for vertices and .

It should be noted that our approach is similar to the generalized belief-propagation algorithm proposed in Ref. Yedidia:yfw01 . It was shown that the belief-propagation algorithm is equivalent to the cluster variation method (CVM) introduced by R. Kikuchi (see Ref. Pelizzola05 for more details). Note that in the model of complex networks in the present work, two motifs can only overlap each other by a single node. They have no joint links. Motifs here play the role of “hyperedges” in contrast to the approach in Ref. Yedidia:yfw01 where clusters are considered as “super-nodes”. Finally, we assumed that networks under consideration have a hypertree-like structure in the infinite size limit.

Iv Critical temperature and critical behavior

In the paramagnetic phase at zero magnetic field, there is no spontaneous magnetization and the effective fields are equal to zero, i.e., . Therefore, we obtain

(44)

for any . This is the only solution of Eqs. (42) and  (LABEL:phi) in the paramagnetic phase. Below a critical temperature , this solution becomes unstable and a non-trivial solution for the function emerges. In this case, a mean value of the effective field ,

(45)

becomes non-zero. Here denotes the thermodynamic average. In the case of a continuous phase transition, tends to zero if the temperature tends to from below. Let use this fact. From Eq. (36) we obtain

(46)

We expand the functions and on the left and right hand sides of this equation in small and , respectively:

(47)

where we define

(48)

is a non-local susceptibility in a spin cluster at zero magnetic field. Note that . Assuming that at the first moment of the function , i.e., , is much larger than higher moments, i.e., , in the leading order we obtain a linear equation,

(49)

where defines an average over the distribution function given by Eq. (14). The function is defined as follows,

(50)

is the total zero-field magnetic susceptibility of the Ising model on a ring of size . Simple calculations give

(51)

where . In the paramagnetic phase, the set of linear equations (49) for parameters has only a trivial solution, . A non-trivial solution appears at a temperature at which

(52)

where the matrix is defined as follows:

(53)

for . Equation (52) determines the critical point of the continuous phase transition. Below , spontaneous effective fields appear, i.e., . Therefore, there is a non-zero spontaneous magnetic moment. If all motifs are loops of equal length , then the critical temperature is determined by the equation

(54)

where is the average branching coefficient for these hyperedges (-loops in this network),

(55)

In particular, if there are only edges, i.e., , then Eq. (54) gives

(56)

This result was found for uncorrelated random complex networks with arbitrary degree distributions dgm02 ; lvvz02 ; Dorogovtsev:dgm08 . In the limit , we find that

(57)
(58)

Figure 4 shows the dependence of the critical temperature on the mean number of nearest neighbors in the networks with the Poissonian distribution of -loop motifs. In this case the average branching coefficient is . Notice also that at we have according to Eqs. (16) and (17). Comparison between Eqs. (57) and (58) shows that at a given mean degree for the Poisson distribution of hyperdegrees the critical temperature is higher than only by 1. This shift is the only effect of finite loops.

If there are only two motifs, for example, loops of size and , then Eq. (52) leads to the equation

(59)
Figure 4: (Color online) Critical temperature of the ferromagnetic Ising model versus the mean number of nearest neighbors in uncorrelated random networks with Poisson distribution of loops of size and the mean number of these loops attached to a vertex. Note that the case is special: there are no loops, and we have the Erdős-Rényi graph. Inset displays versus the mean number of -cliques (dashed lines), and, for comparison, for finite loops of the same size (solid lines). Here we set .

From Eqs. (51) and (54), it follows that if an uncorrelated random hypergraph has a divergent second moment for any , then becomes infinite in the infinite size limit. This result was obtained for ordinary uncorrelated random complex network in Refs. dgm02 ; lvvz02 ; Dorogovtsev:dgm08 .

Another example of complex motifs are -cliques. In this kind of complex networks the clustering coefficient is larger than in networks with loops of size (compare between Eqs. (19) and (20)). Calculating the susceptibility of a cluster of spins on this subgraph, we can find the function in Eq. (50) at ,

(60)

Here we introduced the function

(61)

where is the binomial coefficient. If a network only consists of uncorrelated -cliques then the critical temperature of the ferromagnetic Ising model is given by Eq. (54) with the function Eq. (60). In the limit we obtain an asymptotic result,

(62)

Let us consider the Poisson distribution. In this case we have and the mean degree is . Critical temperatures of networks with cliques and loops are compared in Fig. 4. Comparison between Eqs. (62), (57) and (58) shows that at a given mean degree the critical temperature in Eq. (62) is larger than the critical temperature Eq. (57) only by . This shift is the only effect of clustering. It gives a relative correction of order , because . This result shows that the internal structure of motifs may change the critical temperature of the Ising model although the percolation threshold may be the same (see below).

Thus in the considered random networks the average branching coefficient is crucially important. If , clustering and finite loops lead to a relatively small shift of the critical temperature in comparison to networks without loops but with the same average branching coefficient. However, in networks with a small branching coefficient of motifs, influence of clustering and finite loops is stronger. Indeed, at the critical temperature, Eq. (56), exceeds zero, , if the average branching coefficient (for the Poisson distribution this corresponds to the mean degree ). This is due to the fact that only in this case there is a giant connected component [see Eq. (67)]. In networks with finite loops of size , the percolation threshold from Eq. (67) is . Therefore, the phase transition appears at the average branching coefficient which is much smaller than 1 if (for the Poisson distribution this corresponds to the mean degree ).

Does clustering influence the critical behavior in the network with motifs? In order to answer this question we consider networks that consist of cliques of a given size with the mean number of cliques attached to a node. It is convenient to assume that the distribution function of cliques has the following asymptotic behavior: . In Eq. (LABEL:phi), we use the ”effective medium” approximation introduced in Ref. dgm02 :

(63)

where is the mean effective field acting on a node from an attached -clique. As was shown in Ref. dgm02 , this approximation takes into account the most dangerous highly connected nodes and gives exact critical behavior for random uncorrelated tree-like networks (see also Refs. lvvz02 ; Dorogovtsev:dgm08 ). As a result, we obtain the distribution function of the total field acting on a node from attached -cliques as a function of ,

(64)

Substituting Eqs. (42) and (LABEL:phi) into Eq. (45), we obtain a self consistent equation for the mean effective field ,

(65)

where