Log In Sign Up

Mutation-Driven Follow the Regularized Leader for Last-Iterate Convergence in Zero-Sum Games

by   Kenshi Abe, et al.

In this study, we consider a variant of the Follow the Regularized Leader (FTRL) dynamics in two-player zero-sum games. FTRL is guaranteed to converge to a Nash equilibrium when time-averaging the strategies, while a lot of variants suffer from the issue of limit cycling behavior, i.e., lack the last-iterate convergence guarantee. To this end, we propose mutant FTRL (M-FTRL), an algorithm that introduces mutation for the perturbation of action probabilities. We then investigate the continuous-time dynamics of M-FTRL and provide the strong convergence guarantees toward stationary points that approximate Nash equilibria under full-information feedback. Furthermore, our simulation demonstrates that M-FTRL can enjoy faster convergence rates than FTRL and optimistic FTRL under full-information feedback and surprisingly exhibits clear convergence under bandit feedback.


page 1

page 2

page 3

page 4


Last-Iterate Convergence with Full- and Noisy-Information Feedback in Two-Player Zero-Sum Games

The theory of learning in games is prominent in the AI community, motiva...

Survival of the strictest: Stable and unstable equilibria under regularized learning with partial information

In this paper, we examine the Nash equilibrium convergence properties of...

From Poincaré Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularization

In this paper we investigate the Follow the Regularized Leader dynamics ...

O(T^-1) Convergence of Optimistic-Follow-the-Regularized-Leader in Two-Player Zero-Sum Markov Games

We prove that optimistic-follow-the-regularized-leader (OFTRL), together...

Poincaré-Bendixson Limit Sets in Multi-Agent Learning

A key challenge of evolutionary game theory and multi-agent learning is ...

Learning in nonatomic games, Part I: Finite action spaces and population games

We examine the long-run behavior of a wide range of dynamics for learnin...

Learning in Games with Quantized Payoff Observations

This paper investigates the impact of feedback quantization on multi-age...