Decentralized networks, networks with limited communication between nodes, provide an ideal framework for privacy preserving optimization. They are particularly adapted to the case where agents seek to minimize a global cost function which is separable in the data collected locally by each agent. In such networks, it is typically impossible to efficiently centralize data or to globally aggregate intermediate results: agents can only communicate with their immediate neighbors, often in a completely asynchronous fashion. In a privacy setting, agents never share the raw information they have but rather communicate aggregated information or parameters. Recent work on decentralized optimization focuses on convex optimization: such methods are mostly based on gradient descent [1, 2] or dual averaging [3, 4] and theoretical convergence upper bounds have been established for both synchronous and fully asynchronous computations.
In this paper, we tackle the non-convex problem of topic modelling, where agents have sensitive text data at their disposal that they can not or do not want to share (e.g., text messages, emails, confidential reports). More precisely, we adapt the particular Latent Dirichlet Allocation (LDA)  model to decentralized networks. We combine recent work of  on online inference for latent variable models, which adapts online EM  with local Gibbs sampling in the case of intractable latent variable models (such as LDA) and recent advances on decentralized optimization [3, 4]. The method presented in  is particularly adapted to decentralized framework as it consists in iteratively updating sufficient statistics, which can be done locally. After presenting our DeLeDA (for Decentralized LDA) algorithm, we give a brief sketch of convergence proof. Then, we apply our new method to synthetic datasets and show that our method recovers the same parameters and has similar performance than the online method  after enough iterations.
Latent Dirichlet allocation (LDA) .
Let be the number of documents of a corpus , the number of words in our vocabulary and the number of latent topics in the corpus. Each topic corresponds to a discrete distribution on the words (that is an element of the simplex in dimensions). A hidden discrete distribution over the topics (that is an element of the simplex in dimensions) is attached to each document . LDA is a generative model applied to a corpus of text documents which assumes that each word of the -th document is generated as follows:
For each word :
Choose a topic ,
Choose a word .
In LDA, the inference consists in learning the topic matrix — global parameter — and topic proportions of each document — local variable — from a corpus of documents. We choose to adapt Gibbs Online EM for LDA (G-OEM)  to the decentralized framework. The other inference methods for LDA ([5, 8, 9, 10, 11]) are very close to G-OEM but are less robust in practice (see  for complete analysis).
Gibbs Online EM for LDA .
We consider an exponential family model on random variableswith parameter and with density :
with hidden and observed. G-OEM is an adaptation of the online EM algorithm  that maximizes the likelihood of observed data in the case where the conditional is intractable to compute, such as for LDA. In particular, the G-OEM algorithm iteratively updates the sufficient statistics of the model through the formula at step :
where is the maximum of likelihood for the sufficient statistics , (which is the usual M-step update in the EM algorithm ). For LDA, the expectation in (2) is intractable to compute and is approximated with Gibbs sampling — see  for more details.
In this setting for LDA, an observation is a document . Each observation is associated with the hidden variables , with
. The vectorrepresents the topic proportions of document and is the vector of topic assignments of each word of . The hidden variable is local, attached to one observation . The parameters of the model are global, represented by , where represents the topic matrix and represents the Dirichlet prior on topic proportions.
Let and let be an undirected graph of agents. We consider a setting where each agent aims to optimize a global ojective, without revealing its local observations. In addition, we are interested in the case where communication is possibly asynchronous, there is no global clock ensuring iterations synchrony between nodes. Gossip algorithms  are particularly adapted to this setting, as they rely on peer-to-peer interactions. Initially, each agent stores an iterates ; at each iteration, a node awakes and select one of its neighbor: the two nodes then average their iterates. Then, in the synchronous (resp. asynchronous) setting, all the nodes (resp. the two active nodes) update their iterate according to a local rule, say (2).
When performing decentralized estimationestimating the sample mean, the averaging step allows for a quick uniformization of the network iterates. Indeed, nodes iterates converge to the network average at an exponential rate which depends on some communication property of the graph called the spectral gap: the larger the spectral gap, the faster the convergence. In the case of decentralized optimization, since the network is not conserving its mass during the process — typically, gradients are added —, the convergence is not guaranteed in general. However, provided that at each iteration the mass added to the network is bounded, one can formulate upper bounds on the convergence rate, usually polynomial in the number of iterations.
3 DeLeDA algorithm
In this paper, we aim at combining both G-OEM for LDA and decentralized optimization. Since the G-OEM updates are very similar to stochastic gradient updates, the reasoning of decentralized convex optimization can be extended to G-OEM. In a synchronous setting, our algorithm coined DeLeDA consists in the following: each node initializes its sufficient statistics to . At each iteration, the two awaken nodes average their statistics. Then, every node performs a local update of its statistics using update rule (2) and its own observations. The method is detailed in Algorithm 1.
For , let us define the average iterate . Using Algorithm 1 description and update rule (2), one can see that is obtained from using the G-OEM update rule with all observations of the network. Therefore, convergence derives from G-OEM theoretical guarantees ; in order to prove the convergence of each agent, one only has to show that for all , tends to as the number of iterations grows.
Let us define the iterates matrix and the averaging operator , where and are the nodes awaken at iteration . An iteration of Algorithm 1 can be reformulated as follows:
where is the matrix of expectations in (2). Recursively, one can obtain:
is the second largest eigenvalue of. If the graph is connected and non-bipartite, one can show that . Therefore, when tends towards 0 and provided that is bounded, the expected gap between nodes iterates and the average goes to 0. In practice, this last assumption holds for instance when the Dirichlet parameter is either fixed or lower-bounded, i.e., . Also, as evidenced by inequality (3), one should expect DeLeDA to converge faster on a graph with high spectral gap, as it is proportional to [4, 14].
4 Numerical experiments
We perform experiments on a synthetic graph of =50 nodes. Given these nodes, we consider the complete graph ( edges) and a graph where the edges are generated through a Watts-Strogatz model , with
edges and rewiring probability. We attach 20 documents at each node, generated through LDA generative process, with a vocabulary size and topics. The length of each document is drawn from a Poisson with parameter 10. We compare our algorithm to — centralized — G-OEM applied to the 1000 synthetic documents of the graph with a batch size of 20 documents. In particular, at each iteration of the G-OEM algorithm we choose uniformly at random 20 documents among the 1000 of the total dataset, and update according to the G-OEM process. In practice, we update at each iteration and let fixed, as often done in previous work [8, 11, 16, 17].
We evaluate our method by computing the likelihood on held-out documents, that is for any test document . For LDA, the likelihood is intractable to compute. We approximate with the “left-to-right” evaluation algorithm  applied to each test document. In the following, we present results in terms of log-perplexity, defined as the opposite of the log-likelihood . The lower the log-perplexity, the better the corresponding model. In our experiments, we compute the average test log-perplexity over test documents. More precisely, we display the relative error , where is the average log-perplexity of test documents computed with optimal parameters that generated the data.
We also compare the performance in terms of distance to the optimal topic matrix , which generated the documents. As the performance is invariant to topic permutation in LDA, we compute the following distance between the infered parameter and optimal parameter :
which is invariant by topic permutation (by permutation of rows of ).
Results for asynchrone and synchrone method are presented in Figure 1. We observe that our method converges to the same state than G-OEM after enough iterations. In particular, each agent is able to recover the topic matrix that generated all the documents without any direct access to documents of the other nodes.
As expected, DeLeDA converges faster on a perfectly connected network, even though the “well-connected” design of the Watts-Strogatz network allows it to stay close to the complete graph, perplexity-wise. Finally, the synchronous version of DeLeDA appears to converge slower than the asynchronous one. This is an overfitting issue: synchronous computations lead each node to perform roughly local updates between two averaging steps, whereas asynchronous computations force nodes to update only once between two averaging steps.
-  Angelia Nedic and Asuman Ozdaglar. Distributed subgradient methods for multi-agent optimization. IEEE Transactions on Automatic Control, 54(1):48–61, 2009.
-  Angelia Nedić. Asynchronous broadcast-based convex optimization over a network. Automatic Control, IEEE Transactions on, 56(6):1337–1351, 2011.
-  Alekh Agarwal, Martin J Wainwright, and John C Duchi. Distributed dual averaging in networks. In Advances in Neural Information Processing Systems, pages 550–558, 2010.
-  I. Colin, A. Bellet, J. Salmon, and S. Clémençon. Gossip dual averaging for decentralized optimization of pairwise functions. Proc. ICML, 2016.
-  D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent Dirichlet allocation. JMLR, 3:993–1022, 2003.
-  C. Dupuy and F. Bach. Online but accurate inference for latent variable models with local gibbs sampling. arXiv preprint arXiv:1603.02644, 2016.
-  O. Cappé and E. Moulines. Online EM algorithm for latent data models. Journal of the Royal Statistical Society, 71(3):593–613, 2009.
-  M. D. Hoffman, D. M. Blei, and F. R. Bach. Online learning for latent Dirichlet allocation. Adv. NIPS, 2010.
-  T. L. Griffiths and M. Steyvers. Finding scientific topics. Proceedings of the National Academy of Sciences, 101(suppl 1):5228–5235, 2004.
-  I. Sato, K. Kurihara, and H. Nakagawa. Deterministic single-pass algorithm for LDA. Adv. NIPS, 2010.
-  D. Mimno, M. Hoffman, and D. Blei. Sparse stochastic inference for latent Dirichlet allocation. Proc. ICML, 2012.
-  E. L. Lehmann and G. Casella. Theory of point estimation, volume 31. Springer Science & Business Media, 1998.
-  A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the royal statistical society. Series B (methodological), 39(1):1–38, 1977.
-  Stephen Boyd, Arpita Ghosh, Balaji Prabhakar, and Devavrat Shah. Randomized gossip algorithms. IEEE Trans. Inf. Theory, 52(6):2508–2530, 2006.
-  Duncan J Watts and Steven H Strogatz. Collective dynamics of ‘small-world’networks. Nature, 393(6684):440–442, 1998.
-  S. Patterson and Y. W. Teh. Stochastic gradient Riemannian langevin dynamics on the probability simplex. Adv. NIPS, 2013.
-  Y. Gao, J. Chen, and J. Zhu. Streaming gibbs sampling for LDA model. arXiv preprint arXiv:1601.01142, 2016.
-  H. M. Wallach, I. Murray, R. Salakhutdinov, and D. Mimno. Evaluation methods for topic models. Proc. ICML, 2009.