Parallel Stochastic Asynchronous Coordinate Descent: Tight Bounds on the Possible Parallelism
share
Several works have shown linear speedup is achieved by an asynchronous parallel implementation of stochastic coordinate descent so long as there is not too much parallelism. More specifically, it is known that if all updates are of similar duration, then linear speedup is possible with up to Θ(√(n)/L_res) processors, where L_res is a suitable Lipschitz parameter. This paper shows the bound is tight for essentially all possible values of L_res.
READ FULL TEXT