First-Order Methods with Increasing Iterate Averaging for Solving Saddle-Point Problems
share
First-order methods are known to be among the fastest algorithms for solving large-scale convex-concave saddle-point problems. Algorithms that achieve a theoretical convergence rate on the order of 1/T are known, but these often rely on uniformly averaging iterates in order to get the guaranteed rate. In contrast, using the last iterate has repeatedly been found to perform better in practice, but with no guarantee on convergence rate. In this paper we propose using averaging schemes with increasing weight on recent iterates, which leads to a guaranteed 1/T convergence rate, while capturing the practical performance of using the last iterate. We show this for Chambolle and Pock's primal-dual algorithm, and mirror prox. We present numerical results on computing Nash equilibria in matrix games, competitive equilibria in Fisher markets, and image denoising via total-variation minimization under the ℓ_1 norm. In all cases we find that our averaging schemes lead to much better performance than uniform averaging, and sometimes even better performance than using the last iterate.
READ FULL TEXT