Even Faster Accelerated Coordinate Descent Using Non-Uniform Sampling

12/30/2015
by   Zeyuan Allen-Zhu, et al.
0

Accelerated coordinate descent is widely used in optimization due to its cheap per-iteration cost and scalability to large-scale problems. Up to a primal-dual transformation, it is also the same as accelerated stochastic gradient descent that is one of the central methods used in machine learning. In this paper, we improve the best known running time of accelerated coordinate descent by a factor up to √(n). Our improvement is based on a clean, novel non-uniform sampling that selects each coordinate with a probability proportional to the square root of its smoothness parameter. Our proof technique also deviates from the classical estimation sequence technique used in prior work. Our speed-up applies to important problems such as empirical risk minimization and solving linear systems, both in theory and in practice.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/12/2013

Accelerated Mini-Batch Stochastic Dual Coordinate Ascent

Stochastic dual coordinate ascent (SDCA) is an effective technique for s...
research
11/19/2020

Anderson acceleration of coordinate descent

Acceleration of first order methods is mainly obtained via inertial tech...
research
05/25/2021

SGD with Coordinate Sampling: Theory and Practice

While classical forms of stochastic gradient descent algorithm treat the...
research
05/13/2021

Leveraging Non-uniformity in First-order Non-convex Optimization

Classical global convergence results for first-order methods rely on uni...
research
01/31/2022

L-SVRG and L-Katyusha with Adaptive Sampling

Stochastic gradient-based optimization methods, such as L-SVRG and its a...
research
12/14/2020

ProLab: perceptually uniform projective colour coordinate system

In this work, we propose proLab: a new colour coordinate system derived ...
research
08/03/2018

Coordinate Methods for Accelerating ℓ_∞ Regression and Faster Approximate Maximum Flow

In this paper we provide faster algorithms for approximately solving ℓ_∞...

Please sign up or login with your details

Forgot password? Click here to reset