Linear Last-iterate Convergence for Matrix Games and Stochastic Games

06/16/2020
by   Chung-Wei Lee, et al.
0

Optimistic Gradient Descent Ascent (OGDA) algorithm for saddle-point optimization has received growing attention due to its favorable last-iterate convergence. However, its behavior for simple two-player matrix games is still not fully understood – previous analysis lacks explicit convergence rates, only applies to exponentially small learning rate, or requires additional conditions such as uniqueness of the optimal solution. In this work, we significantly expand the understanding of OGDA, introducing a set of sufficient conditions under which OGDA exhibits concrete last-iterate convergence rates with a constant learning rate. Specifically, we show that matrix games satisfy these conditions and OGDA converges exponentially fast without any additional assumptions. More generally, our conditions hold for smooth bilinear functions and strongly-convex-strongly-concave functions over a constrained set. We provide experimental results to further support our theory. To further demonstrate the significance of our results for matrix games, we greatly generalize the ideas to finite-horizon stochastic/Markov games and provide the first algorithm that simultaneously ensures 1) linear last-iterate convergence when playing against itself and 2) low regret when playing against an arbitrary slowly-changing opponent.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/10/2020

Exponential Convergence of Gradient Methods in Concave Network Zero-sum Games

Motivated by Generative Adversarial Networks, we study the computation o...
research
05/17/2018

Faster Rates for Convex-Concave Games

We consider the use of no-regret algorithms to compute equilibria for pa...
research
06/05/2019

Last-iterate convergence rates for min-max optimization

We study the problem of finding min-max solutions for smooth two-input o...
research
07/08/2020

Stochastic Hamiltonian Gradient Methods for Smooth Games

The success of adversarial formulations in machine learning has brought ...
research
09/30/2020

Gradient Descent-Ascent Provably Converges to Strict Local Minmax Equilibria with a Finite Timescale Separation

We study the role that a finite timescale separation parameter τ has on ...
research
10/26/2020

Tight last-iterate convergence rates for no-regret learning in multi-player games

We study the question of obtaining last-iterate convergence rates for no...
research
03/20/2017

Guaranteed Sufficient Decrease for Variance Reduced Stochastic Gradient Descent

In this paper, we propose a novel sufficient decrease technique for vari...

Please sign up or login with your details

Forgot password? Click here to reset