Learning Trembling Hand Perfect Mean Field Equilibrium for Dynamic Mean Field Games

by   Kiyeob Lee, et al.

Mean Field Games (MFG) are those in which each agent assumes that the states of all others are drawn in an i.i.d. manner from a common belief distribution, and optimizes accordingly. The equilibrium concept here is a Mean Field Equilibrium (MFE), and algorithms for learning MFE in dynamic MFGs are unknown in general due to the non-stationary evolution of the belief distribution. Our focus is on an important subclass that possess a monotonicity property called Strategic Complementarities (MFG-SC). We introduce a natural refinement to the equilibrium concept that we call Trembling-Hand-Perfect MFE (T-MFE), which allows agents to employ a measure of randomization while accounting for the impact of such randomization on their payoffs. We propose a simple algorithm for computing T-MFE under a known model. We introduce both a model-free and a model based approach to learning T-MFE under unknown transition probabilities, using the trembling-hand idea of enabling exploration. We analyze the sample complexity of both algorithms. We also develop a scheme on concurrently sampling the system with a large number of agents that negates the need for a simulator, even though the model is non-stationary. Finally, we empirically evaluate the performance of the proposed algorithms via examples motivated by real-world applications.



page 24


Approximate Equilibrium Computation for Discrete-Time Linear-Quadratic Mean-Field Games

While the topic of mean-field games (MFGs) has a relatively long history...

Model Free Reinforcement Learning Algorithm for Stationary Mean field Equilibrium for Multiple Types of Agents

We consider a multi-agent Markov strategic interaction over an infinite ...

Reinforcement Learning in Non-Stationary Discrete-Time Linear-Quadratic Mean-Field Games

In this paper, we study large population multi-agent reinforcement learn...

Mean Field Equilibrium: Uniqueness, Existence, and Comparative Statics

The standard solution concept for stochastic games is Markov perfect equ...

Multi Type Mean Field Reinforcement Learning

Mean field theory provides an effective way of scaling multiagent reinfo...

Sequential decomposition of graphon mean field games

In this paper, we present a sequential decomposition algorithm to comput...

Mean field voter model on networks and multi-variate beta distribution

The regional characteristics of elections show a strong spatial correlat...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.