Online Learning of Dynamic Parameters in Social Networks

10/01/2013 ∙ by Shahin Shahrampour, et al. ∙ University of Pennsylvania 0

This paper addresses the problem of online learning in a dynamic setting. We consider a social network in which each individual observes a private signal about the underlying state of the world and communicates with her neighbors at each time period. Unlike many existing approaches, the underlying state is dynamic, and evolves according to a geometric random walk. We view the scenario as an optimization problem where agents aim to learn the true state while suffering the smallest possible loss. Based on the decomposition of the global loss function, we introduce two update mechanisms, each of which generates an estimate of the true state. We establish a tight bound on the rate of change of the underlying state, under which individuals can track the parameter with a bounded variance. Then, we characterize explicit expressions for the steady state mean-square deviation(MSD) of the estimates from the truth, per individual. We observe that only one of the estimators recovers the optimal MSD, which underscores the impact of the objective function decomposition on the learning quality. Finally, we provide an upper bound on the regret of the proposed methods, measured as an average of errors in estimating the parameter in a finite time.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

In recent years, distributed estimation, learning and prediction has attracted a considerable attention in wide variety of disciplines with applications ranging from sensor networks to social and economic networks[1, 2, 3, 4, 5, 6]. In this broad class of problems, agents aim to learn the true value of a parameter often called the underlying state of the world. The state could represent a product, an opinion, a vote, or a quantity of interest in a sensor network. Each agent observes a private signal about the underlying state at each time period, and communicates with her neighbors to augment her imperfect observations. Despite the wealth of research in this area when the underlying state is fixed (see e.g.[1, 2, 3, 7]), often the state is subject to some change over time(e.g. the price of stocks)[8, 9, 10, 11]. Therefore, it is more realistic to study models which allow the parameter of interest to vary. In the non-distributed context, such models have been studied in the classical literature on time-series prediction, and, more recently, in the literature on online learning under relaxed assumptions about the nature of sequences [12]. In this paper we aim to study the sequential prediction problem in the context of a social network and noisy feedback to agents.

We consider a stochastic optimization framework to describe an online social learning problem when the underlying state of the world varies over time. Our motivation for the current study is the results of[8] and[9] where authors propose a social learning scheme in which the underlying state follows a simple random walk. However, unlike[8] and[9], we assume a geometric random walk evolution with an associated rate of change. This enables us to investigate the interplay of social learning, network structure, and the rate of state change, especially in the interesting case that the rate is greater than unity. We then pose the social learning as an optimization problem in which individuals aim to suffer the smallest possible loss as they observe the stream of signals. Of particular relevance to this work is the work of Duchi et al. in[13] where the authors develop a distributed method based on dual averaging of sub-gradients to converge to the optimal solution. In this paper, we restrict our attention to quadratic loss functions regularized by a quadratic proximal function, but there is no fixed optimal solution as the underlying state is dynamic. In this direction, the key observation is the decomposition of the global loss function into local loss functions. We consider two decompositions for the global objective, each of which gives rise to a single-consensus-step belief update mechanism. The first method incorporates the averaged prior beliefs among neighbors with the new private observation, while the second one takes into account the observations in the neighborhood as well. In both scenarios, we establish that the estimates are eventually unbiased, and we characterize an explicit expression for the mean-square deviation(MSD) of the beliefs from the truth, per individual. Interestingly, this quantity relies on the whole spectrum of the communication matrix which exhibits the formidable role of the network structure in the asymptotic learning. We observe that the estimators outperform the upper bound provided for MSD in the previous work[8]

. Furthermore, only one of the two proposed estimators can compete with the centralized optimal Kalman Filter

[14] in certain circumstances. This fact underscores the dependence of optimality on decomposition of the global loss function. We further highlight the influence of connectivity on learning by quantifying the ratio of MSD for a complete versus a disconnected network. We see that this ratio is always less than unity and it can get arbitrarily close to zero under some constraints.

Our next contribution is to provide an upper bound for regret of the proposed methods, defined as an average of errors in estimating the parameter up to a given time minus the long-run expected loss due to noise and dynamics alone. This finite-time regret analysis is based on the recently developed concentration inequalities for matrices and it complements the asymptotic statements about the behavior of MSD.

Finally, we examine the trade-off between the network sparsity and learning quality in a microscopic level. Under mild technical constraints, we see that losing each connection has detrimental effect on learning as it monotonically increases the MSD. On the other hand, capturing agents communications with a graph, we introduce the notion of optimal edge as the edge whose addition has the most effect on learning in the sense of MSD reduction. We prove that such a friendship is likely to occur between a pair of individuals with high self-reliance that have the least common neighbors.

2 Preliminaries

2.1 State and Observation Model

We consider a network consisting of a finite number of agents . The agents indexed by seek the underlying state of the world, , which varies over time and evolves according to

(1)

where is a zero mean innovation, which is independent over time with finite variance , and is the expected rate of change of the state of the world, assumed to be available to all agents, and could potentially be greater than unity. We assume the initial state

is a finite random variable drawn independently by the nature. At time period

, each agent receives a private signal , which is a noisy version of , and can be described by the linear equation

(2)

where is a zero mean observation noise with finite variance , and it is assumed to be independent over time and agents, and uncorrelated to the innovation noise. Each agent forms an estimate or a belief about the true value of at time conforming to an update mechanism that will be discussed later. Much of the difficulty of this problem stems from the hardness of tracking a dynamic state with noisy observations, especially when , and communication mitigates the difficulty by virtue of reducing the effective noise.

2.2 Communication Structure

Agents communicate with each other to update their beliefs about the underlying state of the world. The interaction between agents is captured by an undirected graph , where is the set of agents, and if there is a link between agent and agent , then . We let be the set of neighbors of agent , and . Each agent can only communicate with her neighbors, and assigns a weight for any . We also let denote the self-reliance of agent .

Assumption 1.

The communication matrix is symmetric and doubly stochastic, i.e., it satisfies

We further assume the eigenvalues of

are in descending order and satisfy

2.3 Estimate Updates

The goal of agents is to learn in a collaborative manner by making sequential predictions. From optimization perspective, this can be cast as a quest for online minimization of the separable, global, time-varying cost function

(3)

at each time period . One approach to tackle the stochastic learning problem formulated above is to employ distributed dual averaging regularized by a quadratic proximal function[13]. To this end, if agent exploits as the local loss function, she updates her belief as

(4)

while using as the local loss function results in the following update

(5)

where is a constant step size that agents place for their innovation update, and we refer to it as signal weight. Equations (4) and (5) are distinct, single-consensus-step estimators differing in the choice of the local loss function with (4) using only private observations while (5) averaging observations over the neighborhood. We analyze both class of estimators noting that one might expect (5) to perform better than (4) due to more information availability.

Note that the choice of constant step size provides an insight on the interplay of persistent innovation and learning abilities of the network. We remark that agents can easily learn the fixed rate of change by taking ratios of observations, and we assume that this has been already performed by the agents in the past. The case of a changing

is beyond the scope of the present paper. We also point out that the real-valued (rather than vector-valued) nature of the state is a simplification that forms a clean playground for the study of the effects of social learning, effects of friendships, and other properties of the problem.

2.4 Error Process

Defining the local error processes and , at time for agent , as

and stacking the local errors in vectors , respectively, such that

(6)

one can show that the aforementioned collective error processes could be described as a linear dynamical system.

Lemma 2.

Given Assumption 1, the collective error processes and defined in (6) satisfy

(7)

respectively, where

(8)

and

(9)

with being vector of all ones.

Throughout the paper, we let , denote the spectral radius of

, which is equal to the largest singular value of

due to symmetry.

3 Social Learning: Convergence of Beliefs and Regret Analysis

In this section, we study the behavior of estimators (4) and (5) in the mean and mean-square sense, and we provide the regret analysis.

In the following proposition, we establish a tight bound for

, under which agents can achieve asymptotically unbiased estimates using proper signal weight.

Proposition 3 (Unbiased Estimates).

Given the network with corresponding communication matrix satisfying Assumption 1, the rate of change of the social network in (4) and (5) must respect the constraint

to allow agents to form asymptotically unbiased estimates of the underlying state.

Proposition 3 determines the trade-off between the rate of change and the network structure. In other words, changing less than the rate given in the statement of the proposition, individuals can always track with bounded variance by selecting an appropriate signal weight. However, the proposition does not make any statement on the learning quality. To capture that, we define the steady state Mean Square Deviation(MSD) of the network from the truth as follows.

Definition 4 ((Steady State-)Mean Square Deviation).

Given the network with a rate of change which allows unbiased estimation, the steady state of the error processes in (7) is defined as follows

Hence, the (Steady State-)Mean Square Deviation of the network is the deviation from the truth in the mean-square sense, per individual, and it is defined as

Theorem 5 (Msd).

Given the error processes (7) with , the steady state MSD for (4) and (5) is a function of the communication matrix , and the signal weight as follows

(10)

where

(11)

and

(12)

Theorem 5 shows that the steady state MSD is governed by all eigenvalues of contributing to pertaining to the observation noise, while is the penalty incurred due to the innovation noise. Moreover, (5) outperforms (4) due to richer information diffusion, which stresses the importance of global loss function decomposition.

One might advance a conjecture that a complete network, where all individuals can communicate with each other, achieves a lower steady state MSD in the learning process since it provides the most information diffusion among other networks. This intuitive idea is discussed in the following corollary beside a few examples.

Corollary 6.

Denoting the complete, star, and cycle graphs on vertices by , , and , respectively, and denoting their corresponding Laplacians by , , and , under conditions of Theorem 5,

(a)

For , we have

(13)
(b)

For , we have

(14)
(c)

For , where must preserve unbiasedness, we have

(15)
(d)

For , we have

(16)
Proof.

Noting that the spectrum of , and are, respectively[15], , and substituting each case in (10), and taking the limit over , the proof follows immediately. ∎

To study the effect of communication let us consider the estimator (4). Under purview of Theorem 5 and Corollary 6, the ratio of the steady state MSD for a complete network (13) versus a fully disconnected network can be computed as

for . The ratio above can get arbitrary close to zero which, indeed, highlights the influence of communication on the learning quality.

We now consider Kalman Filter(KF)[14] as the optimal centralized counterpart of (5). It is well-known that the steady state KF satisfies a Riccati equation, and when the parameter of interest is scalar, the Riccati equation simplifies to a quadratic with the positive root

Therefore, comparing with the complete graph (16), we have

and the upper bound can be made tight by choosing for . If we should choose an to preserve unbiasedness as well.

On the other hand, to evaluate the performance of estimator (4), we consider the upper bound

(17)

derived in[8], for via a distributed estimation scheme. For simplicity, we assume , and let in (15) be any diminishing function of . Optimizing (13), (14), (15), and (17) over , we obtain

which suggests a noticeable improvement in learning even in the star and cycle networks where the number of individuals and connections are in the same order.

Regret Analysis

We now turn to finite-time regret analysis of our methods. The average loss of all agents in predicting the state, up until time , is

As motivated earlier, it is not possible, in general, to drive this average loss to zero, and we need to subtract off the limit. We thus define regret as

where is from Definition 4. We then have for the spectral norm that

(18)

where we dropped the distinguishing notation between the two estimators since the analysis works for both of them. We, first, state a technical lemma from [16] that we invoke later for bounding the quantity . For simplicity, we assume that magnitudes of both innovation and observation noise are bounded.

Lemma 7.

Let be an independent family of vector valued random variables, and let be a function that maps variables to a self-adjoint matrix of dimension . Consider a sequence of fixed self-adjoint matrices that satisfy

where and range over all possible values of for each index . Letting for all , we have

Theorem 8.

Under conditions of Theorem 5 together with boundedness of noise for some , the regret function defined in (18) satisfies

(19)

with probability at least

.

We mention that results that are similar in spirit have been studied for general unbounded stationary ergodic time series in[17, 18, 19] by employing techniques from the online learning literature. On the other hand, our problem has the network structure and the specific evolution of the hidden state, not present in the above works.

4 The Impact of New Friendships on Social Learning

In the social learning model we proposed, agents are cooperative and they aim to accomplish a global objective. In this direction, the network structure contributes substantially to the learning process. In this section, we restrict our attention to estimator (5), and characterize the intuitive idea that making(losing) friendships can influence the quality of learning in the sense of decreasing(increasing) the steady state MSD of the network.

To commence, letting denote the -th unit vector in the standard basis of , we exploit the negative semi-definite, edge function matrix

(20)

for edge addition(removal) to(from) the graph. Essentially, if there is no connection between agents and ,

(21)

for , corresponds to a new communication matrix adding the edge with a weight to the network , and subtracting from self-reliance of agents and .

Proposition 9.

Let be the network resulted by removing the bidirectional edge with the weight from the network , so and denote the communication matrices associated to and , respectively. Given Assumption 1, for a fixed signal weight the following relationship holds

(22)

as long as is positive semi-definite, and .

Under a mild technical assumption, Proposition 9 suggests that losing connections monotonically increases the MSD, and individuals tend to maintain their friendships to obtain a lower MSD as a global objective. However, this does not elaborate on the existence of individuals with whom losing or making connections could have an immense impact on learning. We bring this concept to light in the following proposition with finding a so-called optimal edge which provides the most MSD reduction, in case it is added to the network graph.

Proposition 10.

Given Assumption 1, a positive semi-definite , and , to find the optimal edge with a pre-assigned weight to add to the network , we need to solve the following optimization problem

(23)

where

(24)

and

are the right eigenvectors of

. In addition, letting

(25)
Proof.

Representing the first order approximation of using definition of in (24), we have for . Based on Theorem 5, we now derive

noting that is from the definition (24). Minimizing is, hence, equivalent to optimization (23) when . Taking into account that is positive semi-definite, for , and which implies , we proceed to the lower bound proof using the definition of and in the statement of the proposition, as follows

Substituting from (24) to above, we have

Using the facts that and according to definition of in (20), and since we are adding a non-existent edge , the lower bound (25) is derived. ∎

Beside posing the optimal edge problem as an optimization, Proposition 10 also provides an upper bound for the best improvement that making a friendship brings to the network. In view of (25), forming a connection between two agents with more self-reliance and less common neighbors, minimizes the lower bound, which offers the most maneuver for MSD reduction.

5 Conclusion

We studied a distributed online learning problem over a social network. The goal of agents is to estimate the underlying state of the world which follows a geometric random walk. Each individual receives a noisy signal about the underlying state at each time period, so she communicates with her neighbors to recover the true state. We viewed the problem with an optimization lens where agents want to minimize a global loss function in a collaborative manner. To estimate the true state, we proposed two methodologies derived from a different decomposition of the global objective. Given the structure of the network, we provided a tight upper bound on the rate of change of the parameter which allows agents to follow the state with a bounded variance. Moreover, we computed the averaged, steady state, mean-square deviation of the estimates from the true state. The key observation was optimality of one of the estimators indicating the dependence of learning quality on the decomposition. Furthermore, defining the regret as the average of errors in the process of learning during a finite time , we demonstrated that the regret function of the proposed algorithms decays with a rate . Finally, under mild technical assumptions, we characterized the influence of network pattern on learning by observing that each connection brings a monotonic decrease in the MSD.

Acknowledgments

We gratefully acknowledge the support of AFOSR MURI CHASE, ONR BRC Program on Decentralized, Online Optimization, NSF under grants CAREER DMS-0954737 and CCF-1116928, as well as Dean’s Research Fund.

References

  • [1] M. H. DeGroot, “Reaching a consensus,” Journal of the American Statistical Association, vol. 69, no. 345, pp. 118–121, 1974.
  • [2] A. Jadbabaie, P. Molavi, A. Sandroni, and A. Tahbaz-Salehi, “Non-bayesian social learning,” Games and Economic Behavior, vol. 76, no. 1, pp. 210–225, 2012.
  • [3] E. Mossel and O. Tamuz, “Efficient bayesian learning in social networks with gaussian estimators,” arXiv preprint arXiv:1002.0747, 2010.
  • [4] O. Dekel, R. Gilad-Bachrach, O. Shamir, and L. Xiao, “Optimal distributed online prediction using mini-batches,”

    The Journal of Machine Learning Research

    , vol. 13, pp. 165–202, 2012.
  • [5] L. Xiao, S. Boyd, and S. Lall, “A scheme for robust distributed sensor fusion based on average consensus,” in Fourth International Symposium on Information Processing in Sensor Networks.   IEEE, 2005, pp. 63–70.
  • [6] S. Kar, J. M. Moura, and K. Ramanan, “Distributed parameter estimation in sensor networks: Nonlinear observation models and imperfect communication,” IEEE Transactions on Information Theory, vol. 58, no. 6, pp. 3575–3605, 2012.
  • [7] S. Shahrampour and A. Jadbabaie, “Exponentially fast parameter estimation in networks using distributed dual averaging,” arXiv preprint arXiv:1309.2350, 2013.
  • [8] D. Acemoglu, A. Nedic, and A. Ozdaglar, “Convergence of rule-of-thumb learning rules in social networks,” in 47th IEEE Conference on Decision and Control, 2008, pp. 1714–1720.
  • [9] R. M. Frongillo, G. Schoenebeck, and O. Tamuz, “Social learning in a changing world,” in Internet and Network Economics.   Springer, 2011, pp. 146–157.
  • [10] U. A. Khan, S. Kar, A. Jadbabaie, and J. M. Moura, “On connectivity, observability, and stability in distributed estimation,” in 49th IEEE Conference on Decision and Control, 2010, pp. 6639–6644.
  • [11] R. Olfati-Saber, “Distributed kalman filtering for sensor networks,” in 46th IEEE Conference on Decision and Control, 2007, pp. 5492–5498.
  • [12] N. Cesa-Bianchi, Prediction, learning, and games.   Cambridge University Press, 2006.
  • [13] J. C. Duchi, A. Agarwal, and M. J. Wainwright, “Dual averaging for distributed optimization: convergence analysis and network scaling,” IEEE Transactions on Automatic Control, vol. 57, no. 3, pp. 592–606, 2012.
  • [14] R. E. Kalman et al., “A new approach to linear filtering and prediction problems,” Journal of basic Engineering, vol. 82, no. 1, pp. 35–45, 1960.
  • [15] M. Mesbahi and M. Egerstedt, Graph theoretic methods in multiagent networks.   Princeton University Press, 2010.
  • [16] J. A. Tropp, “User-friendly tail bounds for sums of random matrices,” Foundations of Computational Mathematics, vol. 12, no. 4, pp. 389–434, 2012.
  • [17] G. Biau, K. Bleakley, L. Györfi, and G. Ottucsák, “Nonparametric sequential prediction of time series,” Journal of Nonparametric Statistics, vol. 22, no. 3, pp. 297–317, 2010.
  • [18] L. Gyorfi and G. Ottucsak, “Sequential prediction of unbounded stationary time series,” IEEE Transactions on Information Theory, vol. 53, no. 5, pp. 1866–1872, 2007.
  • [19] L. Györfi, G. Lugosi et al., Strategies for sequential prediction of stationary time series.   Springer, 2000.

Supplementary Material

Proof of Lemma 2. We subtract (1) from (4) to get

where we used Assumption 1 in the latter step. Replacing from (2) in above, and simplifying using definition of , yields

Using definition (6) to write the above in the matrix form completes the proof for . The proof for follows precisely in the same fashion. ∎

Proof of Proposition 3. We start by the fact that the innovation and observation noise are zero mean, so (7) implies , and . Therefore, for mean stability of the linear equations, the spectral radius of must be less than unity. Considering the expression for from (8), for a fixed we must have

(26)

To maximize the right hand side over , we need to solve the min-max problem

Noting that and are straight lines with negative and positive slopes, respectively, the minimum occurs at the intersection of the two lines. Evaluating the right hand side of (26) at the intersection point , completes the proof.∎

Proof of Theorem 5. We present the proof for by observing that (7)

since the innovation and observation noise are zero mean and uncorrelated. Therefore, letting , since by hypothesis, the steady state satisfies a Lyapunov equation as below

Let represent the Eigen decomposition of . Let also denote the -th eigenvector of corresponding to eigenvalue . Under stability of the solution of the Lyapunov equation is as follows

Therefore, the defined in 4, can be computed as

where we used the fact that , for , and , for any . Taking into account that and , we derive

where the last step is due to the facts that and is one of the eigenvectors of with corresponding eigenvalue , so it is orthogonal to other eigenvectors, i.e., , for . The proof for follows in the same fashion. ∎

Proof of Theorem 8. The closed form solution of the error process (7) is,

which implies