Mutation-Driven Follow the Regularized Leader for Last-Iterate Convergence in Zero-Sum Games

06/18/2022
by   Kenshi Abe, et al.
9

In this study, we consider a variant of the Follow the Regularized Leader (FTRL) dynamics in two-player zero-sum games. FTRL is guaranteed to converge to a Nash equilibrium when time-averaging the strategies, while a lot of variants suffer from the issue of limit cycling behavior, i.e., lack the last-iterate convergence guarantee. To this end, we propose mutant FTRL (M-FTRL), an algorithm that introduces mutation for the perturbation of action probabilities. We then investigate the continuous-time dynamics of M-FTRL and provide the strong convergence guarantees toward stationary points that approximate Nash equilibria under full-information feedback. Furthermore, our simulation demonstrates that M-FTRL can enjoy faster convergence rates than FTRL and optimistic FTRL under full-information feedback and surprisingly exhibits clear convergence under bandit feedback.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/21/2022

Last-Iterate Convergence with Full- and Noisy-Information Feedback in Two-Player Zero-Sum Games

The theory of learning in games is prominent in the AI community, motiva...
research
05/26/2023

A Slingshot Approach to Learning in Monotone Games

In this paper, we address the problem of computing equilibria in monoton...
research
01/12/2021

Survival of the strictest: Stable and unstable equilibria under regularized learning with partial information

In this paper, we examine the Nash equilibrium convergence properties of...
research
07/04/2021

Learning in nonatomic games, Part I: Finite action spaces and population games

We examine the long-run behavior of a wide range of dynamics for learnin...
research
09/26/2022

O(T^-1) Convergence of Optimistic-Follow-the-Regularized-Leader in Two-Player Zero-Sum Markov Games

We prove that optimistic-follow-the-regularized-leader (OFTRL), together...
research
06/27/2023

Semi Bandit Dynamics in Congestion Games: Convergence to Nash Equilibrium and No-Regret Guarantees

In this work, we introduce a new variant of online gradient descent, whi...
research
05/27/2022

Competitive Gradient Optimization

We study the problem of convergence to a stationary point in zero-sum ga...

Please sign up or login with your details

Forgot password? Click here to reset