Provable Fictitious Play for General Mean-Field Games

by   Qiaomin Xie, et al.

We propose a reinforcement learning algorithm for stationary mean-field games, where the goal is to learn a pair of mean-field state and stationary policy that constitutes the Nash equilibrium. When viewing the mean-field state and the policy as two players, we propose a fictitious play algorithm which alternatively updates the mean-field state and the policy via gradient-descent and proximal policy optimization, respectively. Our algorithm is in stark contrast with previous literature which solves each single-agent reinforcement learning problem induced by the iterates mean-field states to the optimum. Furthermore, we prove that our fictitious play algorithm converges to the Nash equilibrium at a sublinear rate. To the best of our knowledge, this seems the first provably convergent single-loop reinforcement learning algorithm for mean-field games based on iterative updates of both mean-field state and policy.



There are no comments yet.


page 1

page 2

page 3

page 4


Actor-Critic Provably Finds Nash Equilibria of Linear-Quadratic Mean-Field Games

We study discrete-time mean-field Markov games with infinite numbers of ...

Fictitious Play for Mean Field Games: Continuous Time Analysis and Applications

In this paper, we deepen the analysis of continuous time Fictitious Play...

Scaling up Mean Field Games with Online Mirror Descent

We address scaling up equilibrium computation in Mean Field Games (MFGs)...

A General Framework for Learning Mean-Field Games

This paper presents a general mean-field game (GMFG) framework for simul...

Generalization in Mean Field Games by Learning Master Policies

Mean Field Games (MFGs) can potentially scale multi-agent systems to ext...

Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time

Reinforcement learning is a powerful tool to learn the optimal policy of...

Concave Utility Reinforcement Learning: the Mean-field Game viewpoint

Concave Utility Reinforcement Learning (CURL) extends RL from linear to ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.