Efficient Distributed Hessian Free Algorithm for Large-scale Empirical Risk Minimization via Accumulating Sample Strategy

10/26/2018
by   Majid Jahani, et al.
0

In this paper, we propose a Distributed Accumulated Newton Conjugate gradiEnt (DANCE) method in which sample size is gradually increasing to quickly obtain a solution whose empirical loss is under satisfactory statistical accuracy. Our proposed method is multistage in which the solution of a stage serves as a warm start for the next stage which contains more samples (including the samples in the previous stage). The proposed multistage algorithm reduces the number of passes over data to achieve the statistical accuracy of the full training set. Moreover, our algorithm in nature is easy to be distributed and shares the strong scaling property indicating that acceleration is always expected by using more computing nodes. Various iteration complexity results regarding descent direction computation, communication efficiency and stopping criteria are analyzed under convex setting. Our numerical results illustrate that the proposed method outperforms other comparable methods for solving learning problems including neural networks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/22/2017

Large Scale Empirical Risk Minimization via Truncated Adaptive Newton Method

We consider large scale empirical risk minimization (ERM) problems, wher...
research
01/01/2015

Communication-Efficient Distributed Optimization of Self-Concordant Empirical Loss

We consider distributed convex optimization problems originated from sam...
research
06/07/2023

Quasi-Newton Updating for Large-Scale Distributed Learning

Distributed computing is critically important for modern statistical ana...
research
02/12/2021

Newton Method over Networks is Fast up to the Statistical Precision

We propose a distributed cubic regularization of the Newton method for s...
research
05/30/2019

Scaling Up Quasi-Newton Algorithms: Communication Efficient Distributed SR1

In this paper, we present a scalable distributed implementation of the s...
research
04/23/2023

Accelerated Doubly Stochastic Gradient Algorithm for Large-scale Empirical Risk Minimization

Nowadays, algorithms with fast convergence, small memory footprints, and...
research
05/23/2018

Approximate Newton-based statistical inference using only stochastic gradients

We present a novel inference framework for convex empirical risk minimiz...

Please sign up or login with your details

Forgot password? Click here to reset