DeepAI AI Chat
Log In Sign Up

On the Optimality of Averaging in Distributed Statistical Learning

by   Jonathan Rosenblatt, et al.
Weizmann Institute of Science

A common approach to statistical learning with big-data is to randomly split it among m machines and learn the parameter of interest by averaging the m individual estimates. In this paper, focusing on empirical risk minimization, or equivalently M-estimation, we study the statistical error incurred by this strategy. We consider two large-sample settings: First, a classical setting where the number of parameters p is fixed, and the number of samples per machine n→∞. Second, a high-dimensional regime where both p,n→∞ with p/n →κ∈ (0,1). For both regimes and under suitable assumptions, we present asymptotically exact expressions for this estimation error. In the fixed-p setting, under suitable assumptions, we prove that to leading order averaging is as accurate as the centralized solution. We also derive the second order error terms, and show that these can be non-negligible, notably for non-linear models. The high-dimensional setting, in contrast, exhibits a qualitatively different behavior: data splitting incurs a first-order accuracy loss, which to leading order increases linearly with the number of machines. The dependence of our error approximations on the number of machines traces an interesting accuracy-complexity tradeoff, allowing the practitioner an informed choice on the number of machines to deploy. Finally, we confirm our theoretical analysis with several simulations.


page 1

page 2

page 3

page 4


A debiased distributed estimation for sparse partially linear models in diverging dimensions

We consider a distributed estimation of the double-penalized least squar...

Distributed linear regression by averaging

Modern massive datasets pose an enormous computational burden to practit...

Fundamental Limits of Ridge-Regularized Empirical Risk Minimization in High Dimensions

Empirical Risk Minimization (ERM) algorithms are widely used in a variet...

Alternating Estimation for Structured High-Dimensional Multi-Response Models

We consider learning high-dimensional multi-response linear models with ...

High-Dimensional Bernoulli Autoregressive Process with Long-Range Dependence

We consider the problem of estimating the parameters of a multivariate B...

Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression

As modern machine learning models continue to advance the computational ...