Stochastic Variance Reduction via Accelerated Dual Averaging for Finite-Sum Optimization
share
In this paper, we introduce a simplified and unified method for finite-sum convex optimization, named Stochastic Variance Reduction via Accelerated Dual Averaging (SVR-ADA). In the nonstrongly convex and smooth setting, SVR-ADA can attain an O(1/n)-accurate solution in nloglog n number of stochastic gradient evaluations, where n is the number of samples; meanwhile, SVR-ADA matches the lower bound of this setting up to a loglog n factor. In the strongly convex and smooth setting, SVR-ADA matches the lower bound in the regime n< O(κ) while it improves the rate in the regime n≫κ to O(nloglog n +nlog(1/(nϵ))/log(n/κ)), where κ is the condition number. SVR-ADA improves complexity of the best known methods without use of any additional strategy such as optimal black-box reduction, and it leads to a unified convergence analysis and simplified algorithm for both the nonstrongly convex and strongly convex settings. Through experiments on real datasets, we also show the superior performance of SVR-ADA over existing methods for large-scale machine learning problems.
READ FULL TEXT