Better Communication Complexity for Local SGD

1 Introduction

Big data optimization problems arising in machine learning and statistics, such as the training of supervised learning models, are routinely solved in a distributed manner on a cluster of compute nodes

Ben-Nun and Hoefler [2018]. Distributed optimization algorithms are typically iterative methods alternating local computations performed on the nodes, and expensive communication steps involving all or a subset of the nodes. Due to the need to solve such problems more efficiently, there has been a lot of recent interest in understanding the trade-offs between communication and computation, a concern which is particularly important in the federated learning setting; see Konečný et al. [2016], Caldas et al. [2018], McMahan et al. [2016].

Minibatch SGD. A popular method for solving unconstrained smooth optimization problems of the form

min x \in R^{d} f (x)

(1)

in situations when the computation of the gradient of $f$ is expensive is minibatch SGD Dekel et al. [2010], Gower et al. [2019]:

x_{t + 1} = x_{t} - \frac{γ_{t}}{M} M \sum m = 1 g_{t}^{m} .

(2)

Here $γ_{t} > 0$ is the stepsize used at time $t$ and $g_{t}^{m}$

is an unbiased estimator of the gradient:

E [g_{t}^{m}] = \nabla f (x_{t})

. In a typical parameter server setup, stochastic gradients

g_{t}^{m}

are computed in parallel by all (or a subset of) nodes

m

, communicated to a parameter server, which subsequently performs the update (2) and communicates it to the nodes, and the process is repeated. As

M

grows, the variance

\frac{1}{M} \sum_{m = 1}^{M} g_{t}^{m}

as an estimator of the gradient

\nabla f (x_{t})

decreases, which leads a decrease in the overall number of communications needed to obtain a solution of sufficient quality.

Local SGD. Note that (2) can equivalently be written in the form

x_{t + 1} = \frac{1}{M} M \sum m = 1 (x_{t} - γ_{t} g_{t}^{m}),

which leads to the alternative interpretation of minibatch SGD as averaging the results of a single SGD step performed by all nodes, initiated from the same starting point $x_{t}$ . This simple observation immediately leads to the natural question: can we gain by performing more than a single step of SGD on each node before averaging? By performing what we hope will be useful additional computation locally on the nodes before expensive aggregation is done, we hope to decrease the number of communication rounds needed. We have just described the local SGD method, formalized as Algorithm 1.

0: Stepsize

γ > 0

, initial vector

x_{0} = x_{0}^{m}

for all

m \in [M]

1: for

t = 0, 1, \dots

2: for

m = 1, \dots, M

3: Sample

g_{t}^{m}

such that

E [g_{t}^{m} ∣ x_{t}^{m}] = \nabla f (x_{t}^{m})

x_{t + 1}^{m} = {\begin{matrix} \frac{1}{m} \sum_{j = 1}^{M} (x_{t}^{j} - γ g_{t}^{j}), & if t = t_{p} for some p \in N x_{t}^{m} - γ g_{t}^{m}, & otherwise. \end{matrix}

5: end for

6: end for

Algorithm 1 Local SGD

2 Contributions

While local SGD has been popular among practitioners for a long time Coppola [2015], McDonald et al. [2010], its theoretical understanding has remained elusive until very recently Zhou and Cong [2018], Stich [2018], Yu et al. [2018], Wang and Joshi [2018], Jiang and Agrawal [2018], Basu et al. [2019] (see Table 1). The history of the methods goes back to the convergence proof in the early work Mangasarian [1995], but a tight convergence rate has been missing since then. Although most existing works focus on analyzing local SGD for smooth and nonconvex $f$ , there are no analyses specialized to the smooth convex case, and only two papers which provide bounds in the smooth strongly convex case.

In this paper we obtain the first result explicitly covering the convex case, and improve dramatically upon the best known communication complexity result in the strongly convex case (see the last row of Table 1). Moreover, unlike previous results in the strongly convex case that depend on a restrictive gradient boundedness assumption, our results do not have this flaw.

Unbounded

gradient

C (T)

f

strongly convex

C (T)

f

weakly convex

C (T)

f

nonconvex

Reference

✓

✗

Ω (T)

Zhou & Cong, 8/2017, Zhou and Cong [2018]

✗

Ω (\sqrt{M T})

✗

Stich, 5/2018, Stich [2018]

✗

Ω (M^{3 / 4} T^{3 / 4})

Yu et al, 7/2018, Yu et al. [2018]

✓

✗

Ω (M^{3 / 2} T^{1 / 2})

Wang & Joshi, 8/2018, Wang and Joshi [2018]

✓

✗

Ω (T)

Jiang & Agarwal, 12/2018, Jiang and Agrawal [2018]

✗

Ω (\sqrt{M T})

✗

Ω (M^{3 / 4} T^{3 / 4})

Basu et al, 6/2019, Basu et al. [2019]

✓

~ Ω (M)

Ω (M^{3 / 2} T^{1 / 2})

✗

THIS WORK, 9/2019

Table 1: Existing theoretical bounds for local SGD.

C (T)

denotes the minimum number of communication steps required each

T

iterations to achieve a linear speedup in the number of nodes

M

An overview of related work on local stochastic gradient methods is given in Table 1.

2.1 Setting and Contributions

In this work we consider minimization problem (1) under the following assumptions:

Assumption 1 (Smoothness and convexity).

We assume $f$ is $L$ -smooth and $μ$ -strongly convex (we allow $μ = 0$ ). That is, for all $x, y \in R^{d}$ we have:

f (y) + ⟨ \nabla f (y), x - y ⟩ + \frac{μ}{2} {∥ x - y ∥}^{2} \leq f (x) \leq f (y) + ⟨ \nabla f (y), x - y ⟩ + \frac{L}{2} {∥ x - y ∥}^{2} .

Assumption 2.

The stochastic gradients $(g_{t}^{m})_{t \geq 0, m \in [M]}$ are unbiased estimates of the true gradient with uniformly bounded variance when conditioned on $x_{t}^{m}$ :

E [g_{t}^{m}] = \nabla f (x_{t}^{m}) and E [{∥ ∥ g_{t}^{m} - \nabla f (x_{t}^{m}) ∥ ∥}^{2}] \leq σ^{2} for all t \geq 0 and m \in [M] .

Note that Assumption 2 is less restrictive than the bounded gradients assumption ( $E [∥ g_{t}^{m} ∥^{2}] \leq G^{2}$ ) used in several previous analysis as shown in Table 1. Under this setting, the main contributions of this paper are:

If $f$ is strongly convex, then by properly choosing stepsizes $γ_{t}$ and taking the average of the local iterates ${^x}_{t}$ , we can obtain $E [{∥ {^x}_{t} - x_{*} ∥}^{2}] \leq ε$ when the total number of iterates $T$ and the total number of communication rounds $C$ are:

$T = ~ Ω (\frac{σ^{2}}{ε M}) and C = Ω (M) .$ (3)

where $~ Ω (\cdot)$ indicates possibly ignoring polylogarithmic factors. This tightens the previous analysis Stich [2018], where $C = Ω (\sqrt{T / M})$ was required.
Furthermore, if $f$ is (possibly weakly) convex, then we can guarantee $E [f (\frac{1}{T} \sum_{i = 0}^{T - 1} {^x}_{i}) - f (x_{*})] \leq ε$ provided that:

$T = Ω (\frac{σ^{4}}{M ε^{2}}) and C = Ω (\frac{\sqrt{T}}{M^{3 / 2}}) .$ (4)
We support our analysis by experiments illustrating the behavior of the algorithm.

3 Convergence Theory

We denote the sequence of time stamps when synchronization happens as $(t_{p})_{p = 1}^{\infty}$ . The average of all local iterates is ${^x}_{t} = \frac{1}{M} \sum_{m = 1}^{M} x_{t}^{m}$ and that of gradients is $g_{t} = \frac{1}{M} \sum_{m = 1}^{M} g_{t}^{m}$ . We define the set $[M] def = {1, 2, \dots, M}$ .

Lemma 1.

Choose a stepsize $γ > 0$ such that $γ \leq \frac{1}{2 L}$ . Under Assumptions 1, and 2 we have that for Algorithm 1 with ${max}_{p} | t_{p} - t_{p + 1} | \leq H$ ,

E [V_{t}] \leq H γ^{2} σ^{2},

(5)

where $V_{t} def = \frac{1}{M} \sum_{m = 1}^{M} {∥ ∥ x_{t}^{m} - {^x}_{t} ∥ ∥}^{2}$ .

Theorem 1.

Suppose that Assumptions 1, and 2 hold with $μ > 0$ . Then combining Lemma 1 with techniques from Stich [2018] we can conclude that for a constant stepsize $γ > 0$ such that $γ \leq \frac{1}{4 L}$ we have for Algorithm 1 with ${max}_{p} | t_{p} - t_{p + 1} | \leq H$ ,

E [{∥ {^x}_{T} - x_{*} ∥}_{T}^{2}] \leq {(1 - γ μ)}^{T} {∥ {^x}_{0} - x_{*} ∥}^{2} + \frac{γ σ^{2}}{μ M} + \frac{2 L γ^{2} H σ^{2}}{μ},

(6)

where ${^x}_{T} = \frac{1}{M} \sum_{m = 1}^{M} x_{T}^{m}$ .

Corollary 1.

Choosing $γ = \frac{1}{μ a}$ , with $a = 4 κ + t$ for $t > 0$ and we take $T = 2 a log a$ steps. Then substituting in (6) and using that $1 - x \leq exp (- x)$ and some algebraic manipulation we can conclude that,

	$E [{∥ {^x}_{T} - x_{*} ∥}_{T}^{2}]$	$\leq \frac{4 {log}^{2} (a)}{T^{2}} {∥ {^x}_{0} - x_{*} ∥}^{2} + \frac{2 σ^{2} log (a)}{μ^{2} M T} + \frac{8 L σ^{2} H {log}^{2} (a)}{μ^{3} T^{2}}$		(7)
				(8)

where $~ O (\cdot)$ ignores logarithmic factors. We see that choosing $H = O (T / M)$ recovers the same $1 / (M T)$ convergence rate of minibatch SGD up to polylogarithmic factors, and the number of communications is then $C = T / H = ~ Ω (M)$ .

Using similar proof techniques, we can show the following result for weakly convex functions:

Theorem 2.

Suppose that Assumptions 1, and 2 hold with $μ = 0$ and that a constant stepsize $γ$ such that $γ \geq 0$ and $γ \leq \frac{1}{4 L}$ is chosen and that Algorithm 1 is run with up to $H$ local iterations, for ${¯ x}_{T} = \frac{1}{T} \sum_{t = 1}^{T} {^x}_{T}$ ,

E [f ({¯ x}_{T}) - f (x_{*})]

\leq \frac{2}{γ T} {∥ x_{0} - x_{*} ∥}_{0}^{2} + \frac{2 γ σ^{2}}{M} + 4 γ^{2} L^{2} σ^{2} H .

(9)

Corollary 2.

Assume that $T \geq M$ . Choosing $γ = \frac{\sqrt{M}}{4 L \sqrt{T}}$ , then substituting in (9) we have,

E [f ({¯ x}_{T}) - f (x_{*})] \leq \frac{8 ∥ x_{0} - x_{*} ∥}{\sqrt{M T}} + \frac{σ^{2}}{2 L \sqrt{M T}} + \frac{σ^{2} M H}{T} .

(10)

We see that choosing $H = O (\sqrt{T} / M^{3 / 2})$ we recover the same $1 / \sqrt{M T}$ convergence rate of minibatch SGD, and the number of communication steps is then $C = T / H = Ω (M^{3 / 2} T^{1 / 2})$ .

4 Experiments

We run experiments on $ℓ_{2}$

regularized logistic regression problem with

M = 20

nodes, each with Intel(R) Xeon(R) Gold 6146 CPU @3.20GHz core. We use the ’a9a’ dataset from the LIBSVM library Chang and Lin [2011] and set

ℓ_{2}

penalty to be

\frac{1}{n}

, where

n

is the dataset size. The code was written in Python using MPI Dalcin et al. [2011].

We ran two experiments, with stepsizes $\frac{1}{L}$ and $\frac{0.05}{L}$ and minibatch size equal 1. In both cases we observe convergence to a neighborhood, although of a different radius. Since we run the experiments on a single machine, the communication is very cheap and there is little gain in time required for convergence. However, the advantage in terms of required communication rounds is self-evident and can lead to significant time improvement under slow communication networks.

Figure 1: Results on regularized logistic regression, ’a9a’ dataset, with stepsize $\frac{1}{L}$ . All local numbers of local iterations converge to a neighborhood within a small number of communication rounds due to large stepsizes.

Figure 2: Results on regularized logistic regression, ’a9a’ dataset, with stepsize $\frac{0.05}{L}$ . With more local iterations, fewer communication rounds are required to get to a neighborhood of the solution.

References

Basu et al. [2019] Debraj Basu, Deepesh Data, Can Karakus, and Suhas Diggavi. Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computations. arXiv:1906.02367, 2019.
Ben-Nun and Hoefler [2018] Tal Ben-Nun and Torsten Hoefler. Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis. arXiv:1802.09941, 2018.
Caldas et al. [2018] Sebastian Caldas, Jakub Konečný, H. Brendan McMahan, and Ameet Talwalkar. Expanding the Reach of Federated Learning by Reducing Client Resource Requirements. arXiv:1812.07210, 2018.
Chang and Lin [2011] Chih-Chung Chang and Chih-Jen Lin.
LIBSVM: A library for support vector machines.
ACM Transactions on Intelligent Systems and Technology (TIST), 2(3):27, 2011.
Coppola [2015] Gregory F. Coppola. Iterative parameter mixing for distributed large-margin training of structured predictors for natural language processing
. PhD thesis, University of Edinburgh, UK, 2015.
Dalcin et al. [2011] Lisandro D. Dalcin, Rodrigo R. Paz, Pablo A. Kler, and Alejandro Cosimo. Parallel distributed computing using Python. Advances in Water Resources, 34(9):1124–1139, 2011.
Dekel et al. [2010] Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, and Lin Xiao. Optimal Distributed Online Prediction using Mini-Batches. arXiv:1012.1367, 2010.
Gower et al. [2019] Robert Mansel Gower, Nicolas Loizou, Xun Qian, Alibek Sailanbayev, Egor Shulgin, and Peter Richtárik. SGD: General Analysis and Improved Rates. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5200–5209, Long Beach, California, USA, 09–15 Jun 2019. PMLR.
Jiang and Agrawal [2018] Peng Jiang and Gagan Agrawal.
A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication.
Advances in Neural Information Processing Systems 31, (NeurIPS):2530–2541, 2018.
Konečný et al. [2016] Jakub Konečný, H. Brendan McMahan, Felix X. Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated Learning: Strategies for Improving Communication Efficiency. In NIPS Private Multi-Party Machine Learning Workshop, 2016.
Mangasarian [1995] L. Mangasarian. Parallel Gradient Distribution in Unconstrained Optimization. SIAM Journal on Control and Optimization, 33(6):1916–1925, 1995.
McDonald et al. [2010] Ryan McDonald, Keith Hall, and Gideon Mann.
Distributed Training Strategies for the Structured Perceptron.
In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, HLT ’10, pages 456–464, Stroudsburg, PA, USA, 2010. Association for Computational Linguistics. ISBN 1-932432-65-5.
McMahan et al. [2016] H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-Efficient Learning of Deep Networks from Decentralized Data. Proceedings of the 20 th International Conference on Artificial Intelligence and Statistics (AISTATS) 2017. JMLR: W&CP volume 54
, 2016.
Stich [2018] Sebastian U. Stich. Local SGD Converges Fast and Communicates Little. arXiv:1805.09767, 2018.
Wang and Joshi [2018] Jianyu Wang and Gauri Joshi. Cooperative SGD: A Unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms. arXiv:1808.07576, 2018.
Yu et al. [2018] Hao Yu, Sen Yang, and Shenghuo Zhu. Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning. arXiv:1807.06629, 2018.
Zhou and Cong [2018] Fan Zhou and Guojing Cong. On the Convergence Properties of a $K$ -step Averaging Stochastic Gradient Descent Algorithm for Nonconvex Optimization. In IJCAI International Joint Conference on Artificial Intelligence, volume 2018-July, pages 3219–3227, 2018.

Appendix A Basic Facts and Notation

We denote the sequence of time stamps when synchronization happens as $(t_{p})_{p = 1}^{\infty}$ . Given stochastic gradients $g_{t}^{1}, g_{t}^{2}, \dots, g_{t}^{M}$ at time $t \geq 0$ we define

g_{t} def = \frac{1}{M} M \sum m = 1 g_{t}^{m},

{¯ g}_{t}^{m} def = E [g_{t}^{m}] = \nabla f (x_{t}^{m}),

{¯ g}_{t} def = E [g_{t}] .

Throughout the proofs, we will use the variance decomposition that holds for any random vector $X$

with finite second moment:

E [∥ X ∥^{2}] = E [∥ X - E [X] ∥^{2}] + ∥ E [X] ∥^{2} .

In particular, its version for vectors with finite number of values gives

(11)

As a consequence of (11) we have that,

Proposition 1 (Jensen’s inequality).

For any convex function $f$ and any vectors $x^{1}, \dots, x^{M}$ we have

f (\frac{1}{M} M \sum m = 1 x^{m}) \leq \frac{1}{M} M \sum m = 1 f (x^{m}) .

We denote the Bregman divergence associated with function $f$ and arbitrary $x, y$ as

D_{f} (x, y) def = f (x) - f (y) - ⟨ \nabla f (y), x - y ⟩ .

Proposition 2.

If $f$ is $L$ -smooth and convex, then for any $x$ and $y$ it holds

∥ \nabla f (x) - \nabla f (y) ∥^{2} \leq 2 L D_{f} (x, y) .

(12)

If $f$ satisfies Assumption 1, then

f (x) + ⟨ \nabla f (y), x - y ⟩ + \frac{μ}{2} {∥ y - x ∥}^{2} \leq f (y) \forall x, y \in R^{d} .

(13)

Appendix B Proof of Lemma 1

Proof.

Let $t \in N$ be such that $t_{p} \leq t < t_{p + 1}$ . Recall that for a time $t$ such that $t_{p} \leq t < t_{p + 1}$ we have $x_{t + 1}^{m} = x_{t}^{m} - γ g_{t}^{m}$ and ${^x}_{t + 1} = {^x}_{t} - γ g_{t}$ . Hence for the expectation conditional on $x_{t}^{1}, x_{t}^{2}, \dots, x_{t}^{M}$ we have:

	$E [{∥ ∥ x_{t + 1}^{m} - {^x}_{t + 1} ∥ ∥}_{t + 1}^{2}]$
		$= {∥ ∥ x_{t}^{m} - {^x}_{t} ∥ ∥}^{2} + γ^{2} E [{∥ ∥ g_{t}^{m} - g_{t} ∥ ∥}_{t}^{2}] - 2 γ ⟨ x_{t}^{m} - {^x}_{t}, \nabla f (x_{t}^{m}) ⟩$
		$+ 2 γ ⟨ x_{t}^{m} - {^x}_{t}, {¯ g}_{t} ⟩ .$

Averaging both sides and letting $V_{t} = \frac{1}{M} \sum_{m} {∥ ∥ x_{t}^{m} - {^x}_{t} ∥ ∥}^{2}$ , we have

	$E [V_{t + 1}]$
		$= V_{t} + \frac{γ^{2}}{M} \sum m E [{∥ ∥ g_{t}^{m} - g_{t} ∥ ∥}_{t}^{2}] - \frac{2 γ}{M} \sum m ⟨ x_{t}^{m} - {^x}_{t}, \nabla f (x_{t}^{m}) ⟩ .$		(14)

Now note that by expanding the square we have,

E [{∥ ∥ g_{t}^{m} - g_{t} ∥ ∥}_{t}^{2}]

(15)

We decompose the first term in the last equality again by expanding the square,

	$E [{∥ ∥ g_{t}^{m} - {¯ g}_{t} ∥ ∥}_{t}^{2}]$

Plugging this into (15) we have,

E [{∥ ∥ g_{t}^{m} - g_{t} ∥ ∥}_{t}^{2}]

Now average over $m$ :

	$\frac{1}{M} \sum m E [{∥ ∥ g_{t}^{m} - g_{t} ∥ ∥}_{t}^{2}]$	$= \frac{1}{M} \sum m E [{∥ ∥ g_{t}^{m} - {¯ g}_{t}^{m} ∥ ∥}^{2}] + \frac{1}{M} \sum m {∥ ∥ {¯ g}_{t}^{m} - {¯ g}_{t} ∥ ∥}^{2} + E [{∥ {¯ g}_{t} - g_{t} ∥}_{t}^{2}]$
		$- 2 E [{∥ {¯ g}_{t} - g_{t} ∥}_{t}^{2}],$

where we used that by definition $\frac{1}{M} \sum_{m = 1}^{M} g_{t}^{m} = g_{t}$ . Hence,

	$\frac{1}{M} \sum m E [{∥ ∥ g_{t}^{m} - g_{t} ∥ ∥}_{t}^{2}]$	$= \frac{1}{M} \sum m E [{∥ ∥ g_{t}^{m} - {¯ g}_{t}^{m} ∥ ∥}^{2}] + \frac{1}{M} \sum m {∥ ∥ {¯ g}_{t}^{m} - {¯ g}_{t} ∥ ∥}^{2} - E [{∥ {¯ g}_{t} - g_{t} ∥}_{t}^{2}]$
		$\leq \frac{1}{M} \sum m E [{∥ ∥ g_{t}^{m} - ¯ g_{t}^{m} ∥ ∥}^{2}] + \frac{1}{M} \sum m {∥ ∥ {¯ g}_{t}^{m} - {¯ g}_{t} ∥ ∥}^{2} .$		(16)

Now note that for the first term in (16) we have by Assumption 2,

E [{∥ ∥ g_{t}^{m} - {¯ g}_{t}^{m} ∥ ∥}^{2}]

(17)

For the second term in (16) we have

{∥ ∥ {¯ g}_{t}^{m} - {¯ g}_{t} ∥ ∥}^{2}

Averaging over $m$ ,

	$\frac{1}{M} M \sum m = 1 {∥ ∥ {¯ g}_{t}^{m} - {¯ g}_{t} ∥ ∥}^{2}$

where we used the fact that $\frac{1}{M} \sum_{m} {¯ g}_{t}^{m} = {¯ g}_{t}$ , which comes from the linearity of expectation. Now we bound ${∥ ∥ {¯ g}_{t}^{m} - \nabla f ({^x}_{t}) ∥ ∥}^{2}$ in the last inequality by smoothness and then use that Jensen’s inequality implies $\sum_{m = 1}^{M} f ({^x}_{t}) - f (x_{t}^{m}) \leq 0$ ,

$\frac{1}{M} \sum m {∥ ∥ {¯ g}_{t}^{m} - \nabla f ({^x}_{t}) ∥ ∥}^{2}$	$= \frac{1}{M} \sum m {∥ ∥ \nabla f (x_{t}^{m}) - \nabla f (^x_{t}) ∥ ∥}^{2}$
	$(???) \leq \frac{1}{M} \sum m 2 L (f ({^x}_{t}) - f (x_{t}^{m}) - ⟨ {^x}_{t} - x_{t}^{m}, \nabla f (x_{t}^{m}) ⟩)$
	$\leq - \frac{2 L}{M} \sum m ⟨ x_{t}^{m} - {^x}_{t}, \nabla f (x_{t}^{m}) ⟩ .$	(18)

Plugging in (18) and (17) into (16) we have,

\frac{1}{M} \sum m E [{∥ ∥ g_{t}^{m} - g_{t} ∥ ∥}_{t}^{2}] \leq σ^{2} - \frac{2 L}{M} \sum m ⟨ x_{t}^{m} - {^x}_{t}, \nabla f (x_{t}^{m}) ⟩ .

(19)

Plugging (19) into (14), we get

$E [V_{t + 1}]$
	$= V_{t} + γ^{2} σ^{2} - \frac{2 γ (1 - γ L)}{M} \sum m ⟨ x_{t}^{m} - {^x}_{t}, \nabla f (x_{t}^{m}) ⟩$	(20)
	$(???) \leq (1 - γ (1 - γ L) μ) V_{t} + γ^{2} σ^{2} .$

Using that $γ \leq \frac{1}{2 L}$ we can conclude,

	$E [V_{t + 1}]$	$\leq (1 - \frac{γ μ}{2}) V_{t} + γ^{2} σ^{2}$
		$\leq V_{t} + γ^{2} σ^{2} .$

Taking expectations and iterating the above inequality,

	$E [V_{t}]$	$\leq E [V_{t_{p}}] + γ^{2} σ^{2} (t - t_{p})$
		$\leq E [V_{t_{p}}] + γ^{2} σ^{2} H .$

It remains to notice that by assumption we have $V_{t_{p}} = 0$ . ∎

Appendix C Two More Lemmas

Lemma 2.

Stich [2018]. Let $(x_{t}^{m})_{t \geq 0}$ be iterates generated by Algorithm 1. Suppose that $f$ satisfies Assumption 1 and that $γ \leq \frac{1}{2 L}$ . Then,

(21)

Proof.

This is Lemma 3.1 in Stich [2018]. ∎

Lemma 3.

Suppose that Assumption 2 holds. Then,

E [{∥ g_{t} - {¯ g}_{t} ∥}_{t}^{2}] \leq \frac{σ^{2}}{M} .

Proof.

This is Lemma 3.2 in Stich [2018]. Because the stochastic gradients $g_{t}^{m}$ are independent we have that the variance of their sum is the sum of their variances, hence

E [{∥ g_{t} - {¯ g}_{t} ∥}_{t}^{2}]

= \frac{1}{M^{2}} E ⎡ ⎣ {∥ ∥ ∥ ∥ M \sum m = 1 g_{t}^{m} - {¯ g}_{t}^{m} ∥ ∥ ∥ ∥}^{2} ⎤ ⎦ = \frac{1}{M^{2}} M \sum m = 1 E [{∥ ∥ g_{t}^{m} - {¯ g}_{t}^{m} ∥ ∥}^{2}] \leq \frac{σ^{2}}{M} .

∎

Appendix D Proof of Theorem 1

Proof.

Combining Lemma 2 and Lemma 3, we have

E [{∥ {^x}_{t + 1} - x_{*} ∥}_{t + 1}^{2}] \leq (1 - γ μ) E [{∥ {^x}_{t} - x_{*} ∥}_{t}^{2}] + \frac{γ^{2} σ^{2}}{M} - \frac{γ}{2} E [D_{f} ({^x}_{t}, x_{*})] + 2 γ L V_{t} .

(22)

Using Lemma 1 we can upper bound the $V_{t}$ term in $(???)$ :

E [{∥ {^x}_{t + 1} - x_{*} ∥}_{t + 1}^{2}] \leq (1 - γ μ) E [{∥ {^x}_{t} - x_{*} ∥}_{t}^{2}] + \frac{γ^{2} σ^{2}}{M} - \frac{γ}{2} E [D_{f} ({^x}_{t}, x_{*})] + 2 γ^{3} L H σ^{2} .

Letting $r_{t + 1} = {^x}_{t + 1} - x_{*}$ and we have,

E [{∥ r_{t + 1} ∥}_{t + 1}^{2}] \leq (1 - γ μ) E [{∥ r_{t} ∥}_{t}^{2}] + \frac{γ^{2} σ^{2}}{M} + 2 γ^{3} L H σ^{2} .

Recursing the above inequality we have,

E [{∥ r_{T} ∥}_{T}^{2}] \leq {(1 - γ μ)}^{T} E [{∥ r_{0} ∥}_{0}^{2}] + (T - 1 \sum t = 0 {(1 - γ μ)}^{t}) (\frac{γ^{2} σ^{2}}{M} + 2 γ^{3} L H σ^{2}) .

Using that $\sum_{t = 0}^{T - 1} {(1 - γ μ)}^{t} \leq \frac{1}{γ μ}$ we have,

E [{∥ r_{T} ∥}_{T}^{2}] \leq {(1 - γ μ)}^{T} E [{∥ r_{0} ∥}_{0}^{2}] + \frac{γ σ^{2}}{μ M} + \frac{2 γ^{2} L H σ^{2}}{μ},

which is the claim of this theorem. ∎

Appendix E Proof of Theorem 2

Proof.

Let $r_{t} = {^x}_{t} - x_{*}$ , then putting $μ = 0$ in Lemma 2 and combining it with Lemma 3, we have

E [{∥ r_{t + 1} ∥}_{t + 1}^{2}] \leq E [{∥ r_{t} ∥}_{t}^{2}] + \frac{γ^{2} σ^{2}}{M} - \frac{γ}{2} E [D_{f} ({^x}_{t}, x_{*})] + 2 γ L V_{t} .

Further using Lemma 1,

E [{∥ r_{t + 1} ∥}_{t + 1}^{2}] \leq E [{∥ r_{t} ∥}_{t}^{2}] + \frac{γ^{2} σ^{2}}{M} - \frac{γ}{2} E [D_{f} ({^x}_{t}, x_{*})] + 2 γ^{3} L H σ^{2} .

Rearranging we have,

Averaging the above equation as $t$ varies between $0$ and $T - 1$ ,

$\frac{γ}{2 T} T - 1 \sum t = 0 E [D_{f} ({^x}_{t}, x_{*})]$	$\leq \frac{1}{T} T - 1 \sum t = 0 E [{∥ r_{t} ∥}_{t}^{2}] - E [{∥ r_{t + 1} ∥}_{t + 1}^{2}] + \frac{1}{T} T - 1 \sum t = 0 (\frac{γ^{2} σ^{2}}{M} + 2 γ^{3} L H σ^{2})$

	$\leq \frac{{∥ r_{0} ∥}^{2}}{T} + \frac{γ^{2} σ^{2}}{M} + 2 γ^{3} L H σ^{2} .$	(23)

By Jensen’s inequality we have $D_{f} ({¯ x}_{T}, x_{*}) \leq \frac{1}{T} \sum_{t = 0}^{T - 1} D_{f} ({^x}_{t}, x_{*})$ . Using this in (23) we have,

Dividing both sides by $γ / 2$ yields the theorem’s claim. ∎

Better Communication Complexity for Local SGD

STL-SGD: Speeding Up Local SGD with Stagewise Communication Period

Random Shuffling Beats SGD after Finite Epochs

The Role of Local Steps in Local SGD

Faster Convergence of Local SGD for Over-Parameterized Models

Optimal Stochastic Strongly Convex Optimization with a Logarithmic Number of Projections

Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization

Byzantine-Resilient High-Dimensional SGD with Local Iterations on Heterogeneous Data

1 Introduction

2 Contributions

2.1 Setting and Contributions

Assumption 1 (Smoothness and convexity).

Assumption 2.

3 Convergence Theory

Lemma 1.

Theorem 1.

Corollary 1.

Theorem 2.

Corollary 2.

4 Experiments

References

Appendix A Basic Facts and Notation

Proposition 1 (Jensen’s inequality).

Proposition 2.

Appendix B Proof of Lemma 1

Proof.

Appendix C Two More Lemmas

Lemma 2.

Proof.

Lemma 3.

Proof.

Appendix D Proof of Theorem 1

Proof.

Appendix E Proof of Theorem 2

Proof.

Better Communication Complexity for Local SGD

Related Research

STL-SGD: Speeding Up Local SGD with Stagewise Communication Period

Random Shuffling Beats SGD after Finite Epochs

The Role of Local Steps in Local SGD

Faster Convergence of Local SGD for Over-Parameterized Models

Optimal Stochastic Strongly Convex Optimization with a Logarithmic Number of Projections

Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization

Byzantine-Resilient High-Dimensional SGD with Local Iterations on Heterogeneous Data

1 Introduction

2 Contributions

2.1 Setting and Contributions

Assumption 1 (Smoothness and convexity).

Assumption 2.

3 Convergence Theory

Lemma 1.

Theorem 1.

Corollary 1.

Theorem 2.

Corollary 2.

4 Experiments

References

Appendix A Basic Facts and Notation

Proposition 1 (Jensen’s inequality).

Proposition 2.

Appendix B Proof of Lemma 1

Proof.

Appendix C Two More Lemmas

Lemma 2.

Proof.

Lemma 3.

Proof.

Appendix D Proof of Theorem 1

Proof.

Appendix E Proof of Theorem 2

Proof.