Stochastic Gradient MCMC with Stale Gradients

1 Introduction

The pervasiveness of big data has made scalable machine learning increasingly important, especially for deep models. A basic technique is to adopt stochastic optimization algorithms

[1], e.g.

, stochastic gradient descent and its extensions

[2]. In each iteration of stochastic optimization, a minibatch of data is used to evaluate the gradients of the objective function and update model parameters (errors are introduced in the gradients, because they are computed based on minibatches rather than the entire dataset; since the minibatches are typically selected at random, this yields the term “stochastic” gradient). This is highly scalable because processing a minibatch of data in each iteration is relatively cheap compared to analyzing the entire (large) dataset at once. Under certain conditions, stochastic optimization is guaranteed to converge to a (local) optima [1]

. Because of its scalability, the minibatch strategy has recently been extended to Markov Chain Monte Carlo (MCMC) Bayesian sampling methods, yielding SG-MCMC

[3, 4, 5].

In order to handle large-scale data, distributed stochastic optimization algorithms have been developed, for example [6], to further improve scalability. In a distributed setting, a cluster of machines with multiple cores cooperate with each other, typically through an asynchronous scheme, for scalability [7, 8, 9]. A downside of an asynchronous implementation is that stale gradients must be used in parameter updates (“stale gradients” are stochastic gradients computed based on outdated parameters, instead of the latest parameters; they are easier to compute in a distributed system, but introduce additional errors relative to traditional stochastic gradients). While some theory has been developed to guarantee the convergence of stochastic optimization with stale gradients [10, 11, 12], little analysis has been done in a Bayesian setting, where SG-MCMC is applied. Distributed SG-MCMC algorithms share characteristics with distributed stochastic optimization, and thus are highly scalable and suitable for large-scale Bayesian learning. Existing Bayesian distributed systems with traditional MCMC methods, such as [13], usually employ stale statistics instead of stale gradients, where stale statistics are summarized based on outdated parameters, e.g., outdated topic distributions in distributed Gibbs sampling [13]. Little theory exists to guarantee the convergence of such methods. For existing distributed SG-MCMC methods, typically only standard stochastic gradients are used, for limited problems such as matrix factorization, without rigorous convergence theory [14, 15, 16].

In this paper, by extending techniques from standard SG-MCMC [17], we develop theory to study the convergence behavior of SG-MCMC with Stale gradients (S $^{2}$ G-MCMC). Our goal is to evaluate the posterior average of a test function $ϕ (x)$ , defined as $¯ ϕ ≜ \int_{X} ϕ (x) ρ (x) d x$ , where $ρ (x)$ is the desired posterior distribution with $x$ the possibly augmented model parameters (see Section 2). In practice, S $^{2}$ G-MCMC generates $L$ samples ${x_{l}}_{l = 1}^{L}$ and uses the sample average ${^ϕ}_{L} ≜ \frac{1}{L} \sum_{l = 1}^{L} ϕ (x_{l})$ to approximate $¯ ϕ$ . We measure how ${^ϕ}_{L}$ approximates $¯ ϕ$ in terms of bias, MSE and estimation variance, defined as $| E {^ϕ}_{L} - ¯ ϕ |$ , $E {({^ϕ}_{L} - ¯ ϕ)}_{L}^{2}$ and $E {({^ϕ}_{L} - E {^ϕ}_{L})}_{L}^{2}$ , respectively. From the definitions, the bias and MSE characterize how accurately ${^ϕ}_{L}$ approximates $¯ ϕ$ , and the variance characterizes how fast ${^ϕ}_{L}$ converges to its own expectation (for a prescribed number of samples $L$ ). Our theoretical results show that while the bias and MSE depend on the staleness of stochastic gradients, the variance is independent of it. In a simple asynchronous Bayesian distributed system with S $^{2}$ G-MCMC, our theory indicates a linear speedup on the decrease of the variance w.r.t. the number of workers used to calculate the stale gradients, while maintaining the same optimal bias level as standard SG-MCMC. We validate our theory on several synthetic experiments and deep neural network models, demonstrating the effectiveness and scalability of the proposed S $^{2}$ G-MCMC framework.

Related Work

Using stale gradients is a standard setup in distributed stochastic optimization systems. Representative algorithms include, but are not limited to, the ASYSG-CON [6] and HOGWILD! algorithms [18], and some more recent developments [19, 20]. Furthermore, recent research on stochastic optimization has been extended to non-convex problems with provable convergence rates [12]. In Bayesian learning with MCMC, existing work has focused on running parallel chains on subsets of data [21, 22, 23, 24], and little if any effort has been made to use stale stochastic gradients, the setting considered in this paper.

2 Stochastic Gradient MCMC

Throughout this paper, we denote vectors as bold lower-case letters, and matrices as bold upper-case letters. For example,

N (m, Σ)

means a multivariate Gaussian distribution with mean

m

and covariance

Σ

. In the analysis we consider algorithms with fixed-stepsizes for simplicity; decreasing-stepsize variants can be addressed similarly as in [17].

The goal of SG-MCMC is to generate random samples from a posterior distribution $p(\mathchar28946\relax|D)∝p(\mathchar28946\relax)∏Ni=1p(di|\mathchar28946\relax)$ , which are used to evaluate a test function. Here $\mathchar28946\relax∈Rn$ represents the parameter vector and $D = {d_{1}, \dots, d_{N}}$ represents the data, $p(\mathchar28946\relax)$ is the prior distribution, and $p(di|\mathchar28946\relax)$ the likelihood for $d_{i}$ . SG-MCMC algorithms are based on a class of stochastic differential equations, called Itô diffusion, defined as

d x_{t}

= F (x_{t}) d t + g (x_{t}) d w_{t},

(1)

where $x \in R^{m}$ represents the model states, typically $x$ augments $\mathchar28946\relax$ such that $\mathchar28946\relax⊆x$ and $n \leq m$ ; $t$ is the time index, $w_{t} \in R^{m}$ is $m$ -dimensional Brownian motion, functions $F : R^{m} \to R^{m}$ and $g : R^{m} \to R^{m \times m}$ are assumed to satisfy the usual Lipschitz continuity condition [25].

For appropriate functions $F$ and $g$ , the stationary distribution, $ρ (x)$ , of the Itô diffusion (1) has a marginal distribution equal to the posterior distribution $p(\mathchar28946\relax|D)$ [26]. For example, denoting the unnormalized negative log-posterior as $U(\mathchar28946\relax)≜−logp(\mathchar28946\relax)−∑Ni=1logp(di|\mathchar28946\relax)$ , the stochastic gradient Langevin dynamic (SGLD) method [3] is based on 1st-order Langevin dynamics, with $x=\mathchar28946\relax$ , and $F(xt)=−∇\mathchar28946\relaxU(\mathchar28946\relax),g(xt)=√2In$ , where $I_{n}$ is the $n \times n$ identity matrix. The stochastic gradient Hamiltonian Monte Carlo (SGHMC) method [4] is based on 2nd-order Langevin dynamics, with $x=(\mathchar28946\relax,q)$ , and $F(xt)=(q−Bq−∇\mathchar28946\relaxU(\mathchar28946\relax)),g(xt)=√2B(000In)$ for a scalar $B > 0$ ; $q$ is an auxiliary variable known as the momentum [4, 5]. Diffusion forms for other SG-MCMC algorithms, such as the stochastic gradient thermostat [5] and variants with Riemannian information geometry [27, 26, 28], are defined similarly.

In order to efficiently draw samples from the continuous-time diffusion (1), SG-MCMC algorithms typically apply two approximations: i) Instead of analytically integrating infinitesimal increments $d t$ , numerical integration over small step size $h$ is used to approximate the integration of the true dynamics. ii) Instead of working with the full gradient $∇\mathchar28946\relaxU(\mathchar28946\relaxlh)$ , a stochastic gradient $∇\mathchar28946\relax~Ul(\mathchar28946\relaxlh)$ , defined as

∇\mathchar28946\relax~Ul(\mathchar28946\relax)≜−∇\mathchar28946\relaxlogp(\mathchar28946\relax)−NJJ∑i=1∇\mathchar28946\relaxlogp(dπi|\mathchar28946\relax),

(2)

is calculated from a minibatch of size $J$ , where ${π_{1}, \dots, π_{J}}$ is a random subset of ${1, \dots, N}$ . Note that to match the time index $t$ in (1), parameters have been and will be indexed by “ $l h$ ” in the $l$ -th iteration.

3 Stochastic Gradient MCMC with Stale Gradients

In this section, we extend SG-MCMC to the stale-gradient setting, commonly met in asynchronous distributed systems [7, 8, 9], and develop theory to analyze convergence properties.

3.1 Stale stochastic gradient MCMC (S $^{2}$ G-Mcmc)

The setting for S $^{2}$ G-MCMC is the same as the standard SG-MCMC described above, except that the stochastic gradient (2) is replaced with a stochastic gradient evaluated with outdated parameter $\mathchar28946\relax(l−τl)h$ instead of the latest version $\mathchar28946\relaxlh$ (see Appendix A for an example):

∇\mathchar28946\relax^Uτl(\mathchar28946\relax)≜−∇\mathchar28946\relaxlogp(\mathchar28946\relax(l−τl)h)−NJJ∑i=1∇\mathchar28946\relaxlogp(dπi|\mathchar28946\relax(l−τl)h),

(3)

where $τ_{l} \in Z^{+}$ denotes the staleness of the parameter used to calculate the stochastic gradient in the $l$ -th iteration. A distinctive difference between S $^{2}$

G-MCMC and SG-MCMC is that stale stochastic gradients are no longer unbiased estimations of the true gradients. This leads to additional challenges in developing convergence bounds, one of the main contributions of this paper.

Input:

xlh=(\mathchar28946\relaxlh,qlh)

∇\mathchar28946\relax^Uτl(\mathchar28946\relax)

τ_{l}

τ

h

B

Output:

x(l+1)h=(\mathchar28946\relax(l+1)h,q(l+1)h)

τ_{l} \leq τ

then

Draw

ζ_{l} \sim N (0, I)

;

q(l+1)h=(1−Bh)qlh−∇\mathchar28946\relax^Uτl(\mathchar28946\relax)h+√2Bhζl

;

\mathchar28946\relax(l+1)h=\mathchar28946\relaxlh+q(l+1)hh

;

end if

Algorithm 1 State update of SGHMC with the stale stochastic gradient

∇\mathchar28946\relax^Uτl(\mathchar28946\relax)

We assume a bounded staleness for all $τ_{l}$ ’s, i.e.,

max l τ_{l} \leq τ

for some constant $τ$ . As an example, Algorithm 1 describes the update rule of the stale-SGHMC in each iteration with the Euler integrator, where the stale gradient $∇\mathchar28946\relax^Uτl(\mathchar28946\relax)$ with staleness $τ_{l}$ is used.

3.2 Convergence analysis

This section analyzes the convergence properties of the basic S $^{2}$ G-MCMC; an extension with multiple chains is discussed in Section 3.3. It is shown that the bias and MSE depend on the staleness parameter $τ$ , while the variance is independent of it, yielding significant speedup in Bayesian distributed systems.

Bias and MSE

In [17], the bias and MSE of the standard SG-MCMC algorithms with a $K$ th order integrator were analyzed, where the order of an integrator reflects how accurately an SG-MCMC algorithm approximates the corresponding continuous diffusion. Specifically, if evolving $x_{t}$ with a numerical integrator using discrete time increment $h$ induces an error bounded by $O (h^{K})$ , the integrator is called a $K$ th order integrator, e.g., the popular Euler method used in SGLD [3] is a 1st-order integrator. In particular, [17] proved the bounds stated in Lemma 1.

Lemma 1 ([17]).

Under standard assumptions (see Appendix B), the bias and MSE of SG-MCMC with a $K$ th-order integrator at time $T = h L$ are bounded as:

	Bias:	$∣ ∣ E {^ϕ}_{L} - ¯ ϕ ∣ ∣ = O (\frac{\sum_{l} ∥ E Δ V_{l} ∥}{L} + \frac{1}{L h} + h^{K})$
	MSE:	$E {({^ϕ}_{L} - ¯ ϕ)}_{L}^{2} = O ⎛ ⎝ \frac{}{\frac{1}{L} \sum_{l} E {∥ Δ V_{l} ∥}_{l}^{2}} L + \frac{1}{L h} + h^{2 K} ⎞ ⎠$

Here $Δ V_{l} ≜ L - {~ L}_{l}$ , where $L$ is the generator of the Itô diffusion (1) defined as

L f (x_{t}) ≜ lim h \to 0^{+} \frac{E [f (x_{t + h})] - f (x_{t})}{h} = (F (x_{t}) \cdot \nabla_{x} + \frac{1}{2} (g (x_{t}) g (x_{t})^{T}) : \nabla_{x} \nabla_{x}^{T}) f (x_{t}),

(4)

for any compactly supported twice differentiable function $f : R^{n} \to R$ , $h \to 0^{+}$ means $h$ approaches zero along the positive real axis. ${~ L}_{l}$ is the same as $L$ except using the stochastic gradient $\nabla {~ U}_{l}$ instead of the full gradient.

We show that the bounds of the bias and MSE of S $^{2}$ G-MCMC share similar forms as SG-MCMC, but with additional dependence on the staleness parameter. In addition to the assumptions in SG-MCMC [17] (see details in Appendix B), the following additional assumption is imposed.

Assumption 1.

The noise in the stochastic gradients is well-behaved, such that: 1) the stochastic gradient is unbiased, i.e., $∇\mathchar28946\relaxU(\mathchar28946\relax)=Eξ∇\mathchar28946\relax~U(\mathchar28946\relax)$ where $ξ$ denotes the random permutation over ${1, \dots, N}$ ; 2) the variance of stochastic gradient is bounded, i.e., $Eξ∥∥U(\mathchar28946\relax)−~U(\mathchar28946\relax)∥∥2≤σ2$ ; 3) the gradient function $∇\mathchar28946\relaxU$ is Lipschitz (so is $∇\mathchar28946\relax~U$ ), i.e., $∥∥∇\mathchar28946\relaxU(x)−∇\mathchar28946\relaxU(y)∥∥≤C∥x−y∥,∀x,y$ .

In the following theorems, we omit the assumption statement for conciseness. Due to the staleness of the stochastic gradients, the term $Δ V_{l}$ in S $^{2}$ G-MCMC is equal to $L - {~ L}_{l - τ_{l}}$ , where ${~ L}_{l - τ_{l}}$ arises from $∇\mathchar28946\relax^Uτl$ . The challenge arises to bound these terms involving $Δ V_{l}$ . To this end, define $f_{l h} ≜ ∥ ∥ x_{l h} - x_{(l - 1) h} ∥ ∥$ , and $ψ$ to be a functional satisfying the Poisson Equation^*^**The existence of a nice $ψ$ is guaranteed in the elliptic/hypoelliptic SDE settings when $x$ is on a torus [25].:

\frac{1}{L} L \sum l = 1 L ψ (x_{l h}) = {^ϕ}_{L} - ¯ ϕ .

(5)

Theorem 2.

After $L$ iterations, the bias of S $^{2}$ G-MCMC with a $K$ th-order integrator is bounded, for some constant $D_{1}$ independent of ${L, h, τ}$ , as:

∣ ∣ E {^ϕ}_{L} - ¯ ϕ ∣ ∣ \leq D_{1} (\frac{1}{L h} + M_{1} τ h + M_{2} h^{K}),

where $M_{1} ≜ {max}_{l} | L f_{l h} | {max}_{l} ∥ E \nabla ψ (x_{l h}) ∥ C$ , $M_{2} ≜ \sum_{k = 1}^{K} \frac{\sum_{l} E {~ L}_{l}^{k + 1} ψ (x_{(l - 1) h})}{(k + 1)! L}$ are constants.

Theorem 3.

After $L$ iterations, the MSE of S $^{2}$ G-MCMC with a $K$ th-order integrator is bounded, for some constant $D_{2}$ independent of ${L, h, τ}$ , as:

E {({^ϕ}_{L} - ¯ ϕ)}_{L}^{2} \leq D_{2} (\frac{1}{L h} + {~ M}_{1} τ^{2} h^{2} + {~ M}_{2} h^{2 K}),

where constants ${~ M}_{1} ≜ {max}_{l} {∥ E \nabla ψ (x_{l h}) ∥}_{l h}^{2} {max}_{l} {(L f_{l h})}_{l h}^{2} C^{2}$ , ${~ M}_{2} ≜ E (\frac{\sum_{l} {~ L}_{l}^{K + 1} ψ (x_{(l - 1) h)}}{L (K + 1)!})^{2}$ .

The theorems indicate that both the bias and MSE depend on the staleness parameter $τ$ . For a fixed computational time, this could possibly lead to unimproved bounds, compared to standard SG-MCMC, when $τ$ is too large, i.e., the terms with $τ$ would dominate, as is the case in the distributed system discussed in Section 4. Nevertheless, better bounds than standard SG-MCMC could be obtained if the decrease of $\frac{1}{L h}$ is faster than the increase of the staleness in a distributed system.

Variance

Next we investigate the convergence behavior of the variance, $Var ({^ϕ}_{L}) ≜ E {({^ϕ}_{L} - E {^ϕ}_{L})}_{L}^{2}$ . Theorem 4 indicates the variance is independent of $τ$ , hence a linear speedup in the decrease of variance is always achievable when stale gradients are computed in parallel. An example is discussed in the Bayesian distributed system in Section 4.

Theorem 4.

After $L$ iterations, the variance of S $^{2}$ G-MCMC with a $K$ th-order integrator is bounded, for some constant $D$ , as:

Var ({^ϕ}_{L}) \leq D (\frac{1}{L h} + h^{2 K}) .

The variance bound is the same as for standard SG-MCMC, whereas $L$ could increase linearly w.r.t. the number of workers in a distributed setting, yielding significant variance reduction. When optimizing the the variance bound w.r.t. $h$ , we get an optimal variance bound stated in Corollary 5.

Corollary 5.

In term of estimation variance, the optimal convergence rate of S $^{2}$ G-MCMC with a $K$ th-order integrator is bounded as: $Var ({^ϕ}_{L}) \leq O (L^{- 2 K / (2 K + 1)})$ .

In real distributed systems, the decrease of $1 / L h$ and increase of $τ$ , in the bias and MSE bounds, would typically cancel, leading to the same bias and MSE level compared to standard SG-MCMC, whereas a linear speedup on the decrease of variance w.r.t. the number of workers is always achievable. More details are discussed in Section 4.

3.3 Extension to multiple parallel chains

This section extends the theory to the setting with $S$ parallel chains, each independently running an S $^{2}$ G-MCMC algorithm. After generating samples from the $S$ chains, an aggregation step is needed to combine the sample average from each chain, i.e., ${^ϕLs}Ms=1$ , where $L_{s}$ is the number of iterations on chain $s$ . For generality, we allow each chain to have different step sizes, e.g., $(h_{s})_{s = 1}^{S}$ . We aggregate the sample averages as ${^ϕ}_{L}^{S} ≜ \sum_{s = 1}^{S} \frac{T_{s}}{T} {^ϕ}_{L_{s}}$ , where $T_{s} ≜ L_{s} h_{s}$ , $T ≜ \sum_{s = 1}^{S} T_{s}$ .

Interestingly, with increasing $S$ , using multiple chains does not seem to directly improve the convergence rate for the bias, but improves the MSE bound, as stated in Theorem 6.

Theorem 6.

Let $T_{m} ≜ {max}_{l} T_{l}$ , $h_{m} ≜ {max}_{l} h_{l}$ , $¯ T = T / S$ , the bias and MSE of $S$ parallel S $^{2}$ G-MCMC chains with a $K$ th-order integrator are bounded, for some constants $D_{1}$ and $D_{2}$ independent of ${L, h, τ}$ , as:

	Bias:	$∣ ∣ E {^ϕ}_{L}^{S} - ¯ ϕ ∣ ∣ \leq D_{1} (\frac{1}{¯ T} + \frac{T_{m}}{¯ T} (M_{1} τ h_{s} + M_{2} h_{s}^{K}))$
	MSE:	$E {({^ϕ}_{L}^{S} - ¯ ϕ)}^{2} \leq D_{2} (\frac{1 - 1 / ¯ T}{T} + \frac{1}{{¯ T}^{2}} + \frac{T_{m}^{2}}{{¯ T}^{2}} (M_{1}^{2} τ^{2} h_{s}^{2} + M_{2}^{2} h_{s}^{2 K})) .$

Assume that $¯ T = T / S$ is independent of the number of chains. As a result, using multiple chains does not directly improve the bound for the bias^†^††It means the bound does not directly relate to low-order terms of $S$ , though constants might be improved.. However, for the MSE bound, although the last two terms are independent of $S$ , the first term decreases linearly with respect to $S$ because $T = ¯ T S$ . This indicates a decreased estimation variance with more chains. This matches the intuition because more samples can be obtained with more chains in a given amount of time.

The decrease of MSE for multiple-chain is due to the decrease of the variance as stated in Theorem 7.

Theorem 7.

The variance of $S$ parallel S $^{2}$ G-MCMC chains with a $K$ th-order integrator is bounded, for some constant $D$ independent of ${L, h, τ}$ , as:

E {({^ϕ}_{L}^{S} - E {^ϕ}_{L}^{S})}^{2} \leq D (\frac{1}{T} + S \sum s = 1 \frac{T_{s}^{2}}{T^{2}} h_{s}^{2 K}) .

When using the same step size for all chains, Theorem 7 gives an optimal variance bound of $O ((\sum_{s} L_{s})^{- 2 K / (2 K + 1)})$ , i.e. a linear speedup with respect to $S$ is achieved.

In addition, Theorem 6 with $τ = 0$ and $K = 1$ provides convergence rates for the distributed SGLD algorithm in [14], i.e., improved MSE and variance bounds compared to the single-server SGLD.

4 Applications to Distributed SG-MCMC Systems

Our theory for S $^{2}$ G-MCMC is general, serving as a basic analytic tool for distributed SG-MCMC systems. We propose two simple Bayesian distributed systems with S $^{2}$ G-MCMC in the following.

Single-chain distributed SG-MCMC

Perhaps the simplest architecture is an asynchronous distributed SG-MCMC system, where a server runs an S $^{2}$ G-MCMC algorithm, with stale gradients computed asynchronously from $W$ workers. The detailed operations of the server and workers are described in Appendix A.

With our theory, now we explain the convergence property of this simple distributed system with SG-MCMC, i.e., a linear speedup w.r.t. the number of workers on the decrease of variance, while maintaining the same bias level. To this end, rewrite $L = W ¯ L$ from Theorems 2 and 3, where $¯ L$ is the average number of iterations on each worker. We can observe from the theorems that when $M_{1} τ h > M_{2} h^{K}$ in the bias and ${~ M}_{1} τ^{2} h^{2} > {~ M}_{2} h^{2 K}$ in the MSE, the terms with $τ$ dominate. Optimizing the bounds with respect to $h$ yields a bound of $O ((τ / W ¯ L)^{1 / 2})$ for the bias, and $O ((τ / W ¯ L)^{2 / 3})$ for the MSE. In practice, we usually observe $τ \approx W$ , making $W$ in the optimal bounds cancels, i.e., the same optimal bias and MSE bounds as standard SG-MCMC are obtained, no theoretical speedup is achieved when increasing $W$ . However, from Corollary 5, the variance is independent of $τ$ , thus a linear speedup on the variance bound can be always obtained when increasing the number of workers, i.e., the distributed SG-MCMC system convergences a factor of $W$ faster than standard SG-MCMC with a single machine. We are not aware of similar conclusions from optimization, because most of the research focuses on the convex setting, thus only variance (equivalent to MSE) is studied.

Multiple-chain distributed SG-MCMC

We can also adopt multiple servers based on the multiple-chain setup in Section 3.3, where each chain corresponds to one server. The detailed architecture is described in Appendix A. This architecture trades off communication cost with convergence rates. As indicated by Theorems 6 and 7, the MSE and variance bounds can be improved with more servers. Note that when only one worker is associated with one server, we recover the setting of $S$ independent servers. Compared to the single-server architecture described above with $S$ workers, from Theorems 2–7, while the variance bound is the same, the single-server arthitecture improves the bias and MSE bounds by a factor of $S$ .

More advanced architectures

More complex architectures could also be designed to reduce communication cost, for example, by extending the downpour [7] and elastic SGD [29] architectures to the SG-MCMC setting. Their convergence properties can also be analyzed with our theory since they are essentially using stale gradients. We leave the detailed analysis for future work.

5 Experiments

Our primal goal is to validate the theory, comparing with different distributed architectures and algorithms, such as [30, 31], is beyond the scope of this paper. We first use two synthetic experiments to validate the theory, then apply the distributed architecture described in Section 4

for Bayesian deep learning. To quantitatively describe the speedup property, we adopt the the

iteration speedup [12], defined as:

{\em iteration speedup}≜\#iterations with a single% workeraverage \#iterations on a worker

, where # is the iteration count when the same level of precision is achieved. This speedup best matches with the theory. We also consider the time speedup, defined as:

\frac{running time for a single worker}{running time for W % worker}

, where the running time is recorded at the same accuracy. It is affected significantly by hardware, thus is not accurately consistent with the theory.

5.1 Synthetic experiments

Figure 1: MSE vs. # iterations ( $L = 500 \times τ$ ) with increasing staleness $τ$ . Resulting in roughly the same MSE.

Impact of stale gradients

A simple Gaussian model is used to verify the impact of stale gradients on the convergence accuracy, with $di∼N(\mathchar28946\relax,1),\mathchar28946\relax∼N(0,1)$ . 1000 data samples ${d_{i}}$ are generated, with minibatches of size 10 to calculate stochastic gradients. The test function is $ϕ(\mathchar28946\relax)≜\mathchar28946\relax2$ . The distributed SGLD algorithm is adopted in this experiment. We aim to verify that the optimal MSE bound $\propto τ^{2 / 3} L^{- 2 / 3}$ , derived from Theorem 3 and discussed in Section 4 (with $W = 1$ ). The optimal stepsize is $h = C τ^{- 2 / 3} L^{- 1 / 3}$ for some constant $C$ . Based on the optimal bound, setting $L = L_{0} \times τ$ for some fixed $L_{0}$ and varying $τ$ ’s would result in the same MSE, which is $\propto L_{0}^{- 2 / 3}$ . In the experiments we set $C = 1 / 30$ , $L_{0} = 500$ , $τ = {1, 2, 5, 10, 15, 20}$ , and average over 200 runs to approximate the expectations in the MSE formula. As indicated in Figure 1, approximately the same MSE’s are obtained after $L_{0} τ$ iterations for different $τ$ values, consistent with the theory. Note since the stepsizes are set to make end points of the curves reach the optimal MSE’s, the curves would not match the optimal MSE curves of $τ^{2 / 3} L^{- 2 / 3}$ in general, except for the end points, i.e., they are lower bounded by $τ^{2 / 3} L^{- 2 / 3}$ .

Convergence speedup of the variance

A Bayesian logistic regression model (BLR) is adopted to verify the variance convergence properties. We use the Adult dataset

^‡^‡‡http://www.csie.ntu.edu.tw/ cjlin/libsvmtools/datasets/binary.html., a9a, with 32,561 training samples and 16,281 test samples. The test function is defined as the standard logistic loss. We average over 10 runs to estimate the expectation

E {^ϕ}_{L}

in the variance. We use the single-server distributed architecture in Section 4, with multiple workers computing stale gradients in parallel. We plot the variance versus the average number of iterations on the workers (

¯ L

) and the running time in Figure 2 (a) and (b), respectively. We can see that the variance drops faster with increasing number of workers. To quantitatively relate these results to the theory, Corollary 5 indicates that

\frac{L_{1}}{L_{2}} = \frac{W_{1}}{W_{2}}

, where

(W_{i}, L_{i})_{i = 1}^{2}

means the number of workers and iterations at the same variance, i.e., a linear speedup is achieved. The iteration speedup and time speedup are plotted in Figure 2 (c), showing that the iteration speedup approximately scales linearly worker numbers, consistent with Corollary 5; whereas the time speedup deteriorates when the worker number is large due to high system latency.

Figure 2: Variance with increasing number of workers.


(a) Variance vs. Iteration $¯ L$	(b) Variance vs. Time	(c) Speedup

5.2 Applications to deep learning

We further test S $^{2}$

G-MCMC on Bayesian learning of deep neural networks. The distributed system is developed based on an MPI (message passing interface) extension of the popular Caffe package for deep learning

[32]. We implement the SGHMC algorithm, with the point-to-point communications between servers and workers handled by the MPICH library.The algorithm is run on a cluster of five machines. Each machine is equipped with eight 3.60GHz Intel(R) Core(TM) i7-4790 CPU cores.

We evaluate S $^{2}$

G-MCMC on the above BLR model and two deep convolutional neural networks (CNN). In all these models, zero mean and unit variance Gaussian priors are employed for the weights to capture weight uncertainties, an effective way to deal with overfitting

[33]. We vary the number of servers

S

among

{1, 3, 5, 7}

, and the number of workers for each server from 1 to 9.

LeNet for MNIST

We modify the standard LeNet to a Bayesian setting for the MNIST dataset.LeNet consists of 2 convolutional layers, 2 max pool layers and 2 ReLU nonlinear layers, followed by 2 fully connected layers

[34]. The detailed specification can be found in Caffe. For simplicity, we use the default parameter setting specified in Caffe, with the additional parameter

B

in SGHMC (Algorithm 1) set to

(1 - m)

, where

m

is the moment variable defined in the SGD algorithm in Caffe.

Cifar10-Quick net for CIFAR10

The Cifar10-Quick net consists of 3 convolutional layers, 3 max pool layers and 3 ReLU nonlinear layers, followed by 2 fully connected layers. The CIFAR-10 dataset consists of 60,000 color images of size 32 $\times$ 32 in 10 classes, with 50,000 for training and 10,000 for testing.Similar to LeNet, default parameter setting specified in Caffe is used.

In these models, the test function is defined as the cross entropy of the softmax outputs ${o_{1}, \dots, o_{N}}$ for test data ${(d_{1}, y_{1}), \dots, (d_{N}, y_{N})}$ with $C$ classes, i.e., $loss = - \sum_{i = 1}^{N} o_{y_{i}} + N log \sum_{c = 1}^{C} e^{o_{c}}$ . Since the theory indicates a linear speedup on the decrease of variance w.r.t. the number of workers, this means for a single run of the models, the loss would converge faster to its expectation with increasing number of workers. The following experiments verify this intuition.

5.2.1 Single-server experiments

We first test the single-server architecture in Section 4 on the three models. Because the expectations in the bias, MSE or variance are not analytically available in these complex models, we instead plot the loss versus average number of iterations ( $¯ L$ defined in Section 4) on each worker and the running time in Figure 3. As mentioned above, faster decrease of the loss with more workers is expected.

For the ease of visualization, we only plot the results with ${1, 2, 4, 6, 9}$ workers; more detailed results are provided in Appendix I. We can see that generally the loss decreases faster with increasing number of workers. In the CIFAR-10 dataset, the final losses of 6 and 9 workers are worst than the one with 4 workers. It shows that the accuracy of the sample average suffers from the increased staleness due to the increased number of workers. Therefore a smaller step size $h$ should be considered to maintain high accuracy when using a large number of workers. Note the 1-worker curves correspond to the standard SG-MCMC, whose loss decreases much slower due to high estimation variance, though in theory it has the same level of bias as the single-server architecture for a given number of iterations (they will converge to the same accuracy).

Figure 3: Testing loss vs. #workers. From left to right, each column corresponds to the a9a, MNIST and CIFAR dataset, respectively. The loss is defined in the text.

5.2.2 Multiple-server experiments

Finally, we test the multiple-servers architecture on the same models. We use the same criterion as the single-server setting to measure the convergence behavior. The loss versus average number of iterations on each worker ( $¯ L$ defined in Section 4) for the three datasets are plotted in Figure 4, where we vary the number of servers among ${1, 3, 5, 7}$ , and use 2 workers for each server. The plots of loss versus time

and using different number of workers for each server are provided in the Appendix. We can see that in the simple BLR model, multiple servers do not seem to show significant speedup, probably due to the simplicity of the posterior, where the sample variance is too small for multiple servers to take effect; while in the more complicated deep neural networks, using more servers results in a faster decrease of the

loss, especially in the MNIST dataset.

Figure 4: Testing loss vs. #servers. From left to right, each column corresponds to the a9a, MNIST and CIFAR dataset, respectively. The loss is defined in the text.

6 Conclusion

We extend theory from standard SG-MCMC to the stale stochastic gradient setting, and analyze the impacts of the staleness to the convergence behavior of an S $^{2}$ G-MCMC algorithm. Our theory reveals that the estimation variance is independent of the staleness, leading to a linear speedup w.r.t. the number of workers, although in practice little speedup in terms of optimal bias and MSE might be achieved due to their dependence on the staleness. We test our theory on a simple asynchronous distributed SG-MCMC system with two simulated examples and several deep neural network models. Experimental results verify the effectiveness and scalability of the proposed S $^{2}$ G-MCMC framework.

Acknowledgements

Supported in part by ARO, DARPA, DOE, NGA, ONR and NSF.

References

[1] L. Bottou, editor. Online algorithms and stochastic approximations. Cambridge University Press, 1998.
[2] L. Bottou. Stochastic gradient descent tricks. Technical report, Microsoft Research, Redmond, WA, 2012.
[3] M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient Langevin dynamics. In ICML, 2011.
[4] T. Chen, E. B. Fox, and C. Guestrin. Stochastic gradient Hamiltonian Monte Carlo. In ICML, 2014.
[5] N. Ding, Y. Fang, R. Babbush, C. Chen, R. D. Skeel, and H. Neven. Bayesian sampling using stochastic gradient thermostats. In NIPS, 2014.
[6] A. Agarwal and J. C. Duchi. Distributed delayed stochastic optimization. In NIPS, 2011.
[7] J. Dean et al. Large scale distributed deep networks. In NIPS, 2012.
[8] T. Chen et al. MXNet: A flexible and efficient machine learning library for heterogeneous distributed systems. (arXiv:1512.01274), Dec. 2015.
[9] M. Abadi et al. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org.
[10] Q. Ho, J. Cipar, H. Cui, J. K. Kim, S. Lee, P. B. Gibbons, G. A. Gibbons, G. R. Ganger, and E. P. Xing. More effective distributed ML via a stale synchronous parallel parameter server. In NIPS, 2013.
[11] M. Li, D. Andersen, A. Smola, and K. Yu. Communication efficient distributed machine learning with the parameter server. In NIPS, 2014.
[12] X. Lian, Y. Huang, Y. Li, and J. Liu. Asynchronous parallel stochastic gradient for nonconvex optimization. In NIPS, 2015.
[13] A. Ahmed, M. Aly, J. Gonzalez, S. Narayanamurthy, and A. J. Smola. Scalable inference in latent variable models. In WSDM, 2012.
[14] S. Ahn, B. Shahbaba, and M. Welling. Distributed stochastic gradient MCMC. In ICML, 2014.
[15] S. Ahn, A. Korattikara, N. Liu, S. Rajan, and M. Welling. Large-scale distributed Bayesian matrix factorization using stochastic gradient MCMC. In KDD, 2015.
[16] U. Simsekli, H. Koptagel, Guldas H, A. Y. Cemgil, F. Oztoprak, and S. Birbil. Parallel stochastic gradient Markov chain Monte Carlo for matrix factorisation models. Technical report, 2015.
[17] C. Chen, N. Ding, and L. Carin. On the convergence of stochastic gradient MCMC algorithms with high-order integrators. In NIPS, 2015.
[18] F. Niu, B. Recht, C. Ré, and S. J. Wright. Hogwild!: A lock-free approach to parallelizing stochastic gradient descent. In NIPS, 2011.
[19] H. R. Feyzmahdavian, A. Aytekin, and M. Johansson. An asynchronous mini-batch algorithm for regularised stochastic optimization. Technical Report arXiv:1505.04824, May 2015.
[20] S. Chaturapruek, J. C. Duchi, and C. Ré. Asynchronous stochastic convex optimization: the noise is in the noise and SGD don’t care. In NIPS, 2015.
[21] S. L. Scott, A. W. Blocker, and F. V. Bonassi. Bayes and big data: The consensus Monte Carlo algorithm. Bayes 250, 2013.
[22] M. Rabinovich, E. Angelino, and M. I. Jordan. Variational consensus Monte Carlo. In NIPS, 2015.
[23] W. Neiswanger, C. Wang, and E. P. Xing. Asymptotically exact, embarrassingly parallel MCMC. In UAI, 2014.
[24] X. Wang, F. Guo, K. Heller, and D. Dunson. Parallelizing MCMC with random partition trees. In NIPS, 2015.
[25] J. C. Mattingly, A. M. Stuart, and M. V. Tretyakov. Construction of numerical time-average and stationary measures via Poisson equations. SIAM Journal on Numerical Analysis, 48(2):552–577, 2010.
[26] Y. A. Ma, T. Chen, and E. B. Fox. A complete recipe for stochastic gradient MCMC. In NIPS, 2015.
[27] S. Patterson and Y. W. Teh. Stochastic gradient Riemannian Langevin dynamics on the probability simplex. In NIPS, 2013.
[28] C. Li, C. Chen, D. Carlson, and L. Carin. Preconditioned stochastic gradient Langevin dynamics for deep neural networks. In AAAI, 2016.
[29] S. Zhang, A. E. Choromanska, and Y. Lecun. Deep learning with elastic averaging SGD. In NIPS, 2015.
[30] Y. W. Teh, L. Hasenclever, T. Lienart, S. Vollmer, S. Webb, B. Lakshminarayanan, and C. Blundell. Distributed Bayesian learning with stochastic natural-gradient expectation propagation and the posterior server. Technical Report arXiv:1512.09327v1, December 2015.
[31] R. Bardenet, A. Doucet, and C. Holmes. On Markov chain Monte Carlo methods for tall data. Technical Report arXiv:1505.02827, May 2015.
[32] T. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.
[33] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural networks. In ICML, 2015.
[34] A. Krizhevshy, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
[35] S. J. Vollmer, K. C. Zygalakis, and Y. W. Teh. (Non-)asymptotic properties of stochastic gradient Langevin dynamics. Technical Report arXiv:1501.00438, University of Oxford, UK, January 2015.

Appendix A A simple Bayesian distributed system based on S $^{2}$ G-Mcmc

We provide the detailed architecture of the simple Bayesian distributed system described in Section 4. We put the single-chain and multiple-chain distributed SG-MCMCs into a unified framework. Suppose there are $S$ servers and $W$ workers, the one with $S = 1$ corresponds to the single-chain distributed SG-MCMC, whereas the one with $S > 1$ corresponds to the multiple-chain distributed SG-MCMC. The servers and workers are responsible for the following tasks:

Each worker runs independently and communicates with a specific server. They are responsible for computing the stochastic gradients^§^§§This is the most expensive part in an SG-MCMC algorithm. of the parameter given by the server. Once the stochastic gradient is computed, the worker sends it to its assigned server and receive a new parameter sample from the server.
Each server independently maintains its own state vector and timestamp. At the $l$ -th timestamp^¶^¶¶Each server is equipped with a timestamp because they are independent with each other., it receives a stale stochastic gradient $∇\mathchar28946\relax^Uτl(\mathchar28946\relax)≜∇\mathchar28946\relax~U(\mathchar28946\relax(l−τl)h)$ from worker $w$ , updates the state vector $x_{l h}$ to $x_{(l + 1) h}$ and increments the timestamp, then sends the new parameter sample $\mathchar28946\relax(l+1)h$ to worker $w$ .

The sending and receiving in the servers and workers are performed asynchronously, enabling minimum communication cost and latency between the servers and workers. At testing, all the samples from the servers are collected and applied to a test function. Apparently, the training time using multiple servers is basically the same as using a single server because the sampling in different servers is independent. Figure 5 depicts the architecture of the proposed Bayesian distributed framework. Algorithm 2 details the algorithm on the servers and workers.

Server

Output:

{x_{h}, \dots, x_{L h}}

Initialize

x_{0} \in R^{m}

;

Send

\mathchar28946\relax0

to all assigned workers;

for

l = 0, 1, \dots, L - 1

Receive a stale stochastic gradient

\nabla {~ U}_{(l - τ_{l}) h}

from a worker

w

Update

x_{l h}

x_{(l + 1) h}

using

\nabla {~ U}_{(l - τ_{l}) h}

. (*)

Send

\mathchar28946\relax(l+1)h

to the worker

w

end for

Worker

repeat

Receive

\mathchar28946\relaxlh

from server

s

Compute

\nabla {~ U}_{l h}

with a minibatch.

Send

\nabla {~ U}_{l h}

to server

s

until

s t o p

Algorithm 2 Asynchronous Distributed SG-MCMC

The update rule (*) of the state vector in Algorithm 2 depends on which SG-MCMC algorithm is employed. For instance, Algorithm 1 describes the update rule of the SGHMC with a 1st-order Euler integrator.

Appendix B Assumptions

First, following [25], we will need to assume the corresponding SDE of SG-MCMC to be either elliptic or hypoelliptic. The ellipticity/hypoellipticity describes whether the Brownian motion is able to spread over the whole parameter space. The SDE of the SGLD is elliptic, while for other SG-MCMC algorithms such as the SGHMC, the hypoellipticity assumption is usually reasonable. When the domain $x$ is on the torus, the ellipticity and hypoellipticity of an SDE guarantees the existence of a nice solution for the Poisson equation (5). The assumption is summarized in Assumption 2.

Figure 5: Architecture of the proposed Bayesian distributed framework. In the multi-server case, the dash lines on the servers indicate a simple averaging operation for testing, otherwise the servers are independent. Section 3.3 provides more details.

Assumption 2.

The corresponding SDE of a SG-MCMC algorithm is either elliptic or hypoelliptic^∥^∥∥The SDE of the SGLD can be verified to be elliptic. For other SG-MCMC algorithms such as the SGHMC, the hypoellipticity assumption is usually reasonable, see [25] on how to verify hypoellipticity of an SDE..

When $x$ is extended to the domain of $R^{p}$ for some integer $p > 0$ , we need some assumptions on the solution of the Poisson equation (5). Note (5) can be equivalently written in an integration form [35] using Itô’s formula:

				(6)
	$=$	$\frac{1}{t} (ψ (x_{t}) - ψ (x_{0})) - \frac{1}{t} \int_{0}^{t} \nabla ψ (x_{s}) \cdot g (x_{s}) d w_{s} .$

Intuitively, $ψ$ needs to be bounded if the discrepancy between ${^ϕ}_{L}$ and $¯ ϕ$ were to be bounded. This is satisfied if the SDE is defined in a bounded domain [25]. In the unbounded domain as for SG-MCMC algorithms, it turns out the following boundedness assumptions on $ψ$ suffice [17].

Assumption 3.

1) $ψ$ and its up to 3rd-order derivatives, $D^{k} ψ$ , are bounded by a function $V$ , i.e., $∥ D^{k} ψ ∥ \leq C_{k} V^{p_{k}}$ for $k = (0, 1, 2, 3)$ , $C_{k}, p_{k} > 0$ . 2) the expectation of $V$ on ${x_{l h}}$ is bounded: ${sup}_{l} E V^{p} (x_{l h}) < \infty$ . 3) $V$ is smooth such that ${sup}_{s \in (0, 1)} V^{p} (s x + (1 - s) y) \leq C (V^{p} (x) + V^{p} (y))$ , $\forall x, y, p \leq max {2 p_{k}}$ for some $C > 0$ .

Furthermore, in our proofs the expectation of a function under a diffusion needs to be expanded in a Taylor expansion style, e.g., $E ϕ (x_{t}) = \sum_{i = 0}^{ℓ} \frac{t^{i}}{i!} L^{i} ϕ (x_{0}) + t^{ℓ + 1} r_{ℓ, F, ϕ} (x_{0})$ by using Kolmogorov’s backward equation. To ensure the remainder term $r_{ℓ, F, ϕ} (x_{0})$ to be bounded, it suffices to make the following assumption on the smoothness and boundedness of $F (x)$ [35, 17].

Assumption 4.

$F (x)$ is infinitely differentiable with bounded derivatives of any order; and $| F (x) | \leq A (1 + | x |^{s})$ for some integer $s > 0$ and $A > 0$ .

Appendix C Notation

For simplicity, we will simplify some notation used in the proof as follows:

	$∇\mathchar28946\relax~Ul(\mathchar28946\relaxlh)≜∇\mathchar28946\relax~Ulh≜~Glh$
	$∇\mathchar28946\relaxUl(\mathchar28946\relaxlh)≜∇\mathchar28946\relaxUlh≜Glh$
	$ψ (X_{l h}) ≜ ψ_{l h}$

Appendix D Proof of Theorem 2

In S $^{2}$ G-MCMC, for the $l$ -th iteration, suppose a stochastic gradient with a staleness $τ_{l}$ is used, e.g., ${~ G}_{(l - τ_{l}) h}$ . First, we will bound the difference between ${~ G}_{(l - τ_{l}) h}$ and the stochastic gradient at the $l$ -th iteration ${~ G}_{l h}$ , by using the Lipschitz property of ${~ G}_{l h}$ , with the following lemma.

Lemma 8.

Let $f_{l h} ≜ ∥ ∥ x_{l h} - x_{(l - 1) h} ∥ ∥$ , the expected difference between ${~ G}_{(l - τ_{l}) h}$ and ${~ G}_{l h}$ is bounded by:

∥ ∥ E ({~ G}_{(l - τ_{l}) h} - {~ G}_{l h}) ∥ ∥ = l - 1 max i = l - τ_{l} | L_{i} f_{i h} | C τ h + O (h^{2}),

(7)

where the expectation is taken over the randomness of the SG-MCMC algorithm, e.g., the randomness from stochastic gradients and the injected Gaussian noise.

Proof.

Note the randomness of ${~ G}_{l h}$ comes from two sources, the injected Gaussian noise and the stochastic gradient noise. We denote the expectations with respect to these two randomness as $E_{ζ}$ and $E_{g}$ , respectively. The whole expectation thus can be decomposed as $E = E_{ζ} E_{g}$ .

Applying the Lipschitz property of ${~ G}_{l h}$ , we have

	$∥ ∥ E ({~ G}_{(l - τ_{l}) h} - {~ G}_{l h}) ∥ ∥ = ∥ ∥ E_{ζ} (G_{(l - τ_{l}) h} - G_{l h}) ∥ ∥$

	$≤CEζ∥∥(\mathchar28946\relax(l−τl)h−\mathchar28946\relaxlh)∥∥$


	$\leq C l - 1 \sum i = l - τ_{l} E_{ζ} ∥ ∥ x_{(i + 1) h} - x_{i h} ∥ ∥$

From the definition of $K$ th-order integrator, i.e., $E_{ζ} f (x_{l h}) = e^{{~ L}_{l} h} f (x_{(l - 1) h}) + O (h^{K + 1})$ , if we let

f (x_{l h}) = ∥ ∥ x_{l h} - x_{(l - 1) h} ∥ ∥ ≜ f_{l h},

where $x_{(l - 1) h}$ is the starting point in the $l$ -th iteration, and note that

f (x_{(l - 1) h}) = 0 .

We have

	$C l - 1 \sum i = l - τ_{l} E_{ζ} ∥ ∥ x_{(i + 1) h} - x_{i h} ∥ ∥ ≜ C l - 1 \sum i = l - τ_{l} E_{ζ} f (x_{l h})$	(8)
$\leq$	$C l - 1 \sum i = l - τ_{l} (e^{L_{i} h} f (x_{(i - 1) h}) + O (h^{K + 1}))$	(9)
$\leq$	$C l - 1 \sum i = l - τ_{l} \| L_{i} f_{i h} \| h + O (h^{2})$	(10)
$\leq$	$l - 1 max i = l - τ_{l} \| L_{i} f_{i h} \| C τ h + O (h^{2}),$

where (10) is obtained by expanding the exponential operator and the assumption that the high order terms are bounded. ∎

Now we proceed to prove Theorem 2. The basic technique follows [17], thus we skip some derivations for some steps.

Proof of Theorem 2.

Before the proof, let us first define some notation. First, define the operator $Δ V_{l}$ for each $l$ as a differential operator as for any function $ψ$ :

Δ V_{l} ψ ≜ ({~ G}_{l - τ_{l}} - G_{l}) \cdot \nabla_{p} ψ .

Second, define the local generator, ${~ L}_{l}$ , for an Itô diffusion, where the true gradient in (1) is replaced with the stochastic gradient from the $l$ -th iteration, i.e., ${~ L}_{l} f (X_{t}) ≜$

for a compactly supported twice differentiable function $f$ , where ${~ F}_{l}$ is the same as $F$ but with the full gradient $G_{l h}$ replaced with the stochastic gradient ${~ G}_{l h}$ . Based on these definitions, we have

{~ L}_{l} = L + Δ V_{l} .

Following [17], for an SG-MCMC with a $K$ th-order integrator, and a test function $ϕ$ , we have:

	$E$	$[ψ (x_{l h})] = (I + h {~ L}_{l}) ψ (x_{(l - 1) h})$		(11)
		$+ K \sum k = 2 \frac{h^{k}}{k!} {~ L}_{l}^{k} ψ (x_{(l - 1) h}) + O (\frac{h^{K + 1}}{(K + 1)!} {~ L}_{l}^{K + 1} ψ_{(l - 1) h}),$

where $I$ is the identity map. Sum over $l = 1, \dots, L$ in (11), take expectation on both sides, and use the relation ${~ L}_{l} = L + Δ V_{l}$ to expand the first order term. We obtain

	$L \sum l = 1 E$
		$+ h L \sum l = 1 E [L ψ (x_{(l - 1) h})] + h L \sum l = 1 E [Δ V_{l} ψ (x_{(l - 1) h})]$
		$+ K \sum k = 2 \frac{h^{k}}{k!} L \sum l = 1 E [{~ L}_{l}^{k} ψ (x_{(l - 1) h})]$
		$+ O (\frac{h^{K + 1}}{(K + 1)!} \sum l E {~ L}_{l}^{K + 1} ψ_{(l - 1) h}) .$

Divide both sides by $L h$ , use the Poisson equation (5), and reorganize terms. We have:

	$E [\frac{1}{L} \sum l ϕ (x_{l h}) - ¯ ϕ] = \frac{1}{L} L \sum l = 1 E [L ψ (x_{(l - 1) h})]$	(12)
$=$	$\frac{1}{L h} (E [ψ (x_{l h})] - ψ (x_{0})) - \frac{1}{L} \sum l E [Δ V_{l} ψ (x_{(l - 1) h})]$
	$- K \sum k = 2 \frac{h^{k - 1}}{k! L} L \sum l = 1 E [{~ L}_{l}^{k} ψ (x_{(l - 1) h})] + O (\frac{h^{K}}{(K + 1)! L} \sum l E {~ L}_{l}^{K + 1} ψ_{(l - 1) h})$

According to [17], the term $\sum_{l} E [{~ L}_{l}^{k} ψ (x_{(l - 1) h})]$ is bounded by $\sum_{l} E [{~ L}_{l}^{k} ψ (X_{(l - 1) h})]$

= O (\frac{1}{h} + h^{K - k}

Stochastic Gradient MCMC with Stale Gradients

Authors

Related Research

Variance reduction for distributed stochastic gradient MCMC

A Convergence Analysis for A Class of Practical Variance-Reduction Stochastic Gradient MCMC

On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators

Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning

Improving Sampling Accuracy of Stochastic Gradient MCMC Methods via Non-uniform Subsampling of Gradients

Stochastic Gradient MCMC for State Space Models

Deep Latent Dirichlet Allocation with Topic-Layer-Adaptive Stochastic Gradient Riemannian MCMC

This week in AI

1 Introduction

Related Work

2 Stochastic Gradient MCMC

3 Stochastic Gradient MCMC with Stale Gradients

3.1 Stale stochastic gradient MCMC (S2G-Mcmc)

3.2 Convergence analysis

Bias and MSE

Lemma 1 ([17]).

Assumption 1.

Theorem 2.

Theorem 3.

Variance

Theorem 4.

Corollary 5.

3.3 Extension to multiple parallel chains

Theorem 6.

Theorem 7.

4 Applications to Distributed SG-MCMC Systems

Single-chain distributed SG-MCMC

Multiple-chain distributed SG-MCMC

More advanced architectures

5 Experiments

5.1 Synthetic experiments

Impact of stale gradients

Convergence speedup of the variance

5.2 Applications to deep learning

LeNet for MNIST

Cifar10-Quick net for CIFAR10

5.2.1 Single-server experiments

5.2.2 Multiple-server experiments

6 Conclusion

Acknowledgements

References

Appendix A A simple Bayesian distributed system based on S2G-Mcmc

Appendix B Assumptions

Assumption 2.

Assumption 3.

Assumption 4.

Appendix C Notation

Appendix D Proof of Theorem 2

Lemma 8.

Proof.

Proof of Theorem 2.

3.1 Stale stochastic gradient MCMC (S $^{2}$ G-Mcmc)

Appendix A A simple Bayesian distributed system based on S $^{2}$ G-Mcmc