On the Statistical Efficiency of Mean Field Reinforcement Learning with General Function Approximation

05/18/2023
by   Jiawei Huang, et al.
0

In this paper, we study the statistical efficiency of Reinforcement Learning in Mean-Field Control (MFC) and Mean-Field Game (MFG) with general function approximation. We introduce a new concept called Mean-Field Model-Based Eluder Dimension (MBED), which subsumes a rich family of Mean-Field RL problems. Additionally, we propose algorithms based on Optimistic Maximal Likelihood Estimation, which can return an ϵ-optimal policy for MFC or an ϵ-Nash Equilibrium policy for MFG, with sample complexity polynomial w.r.t. relevant parameters and independent of the number of states, actions and the number of agents. Notably, our results only require a mild assumption of Lipschitz continuity on transition dynamics and avoid strong structural assumptions in previous work. Finally, in the tabular setting, given the access to a generative model, we establish an exponential lower bound for MFC setting, while providing a novel sample-efficient model elimination algorithm to approximate equilibrium in MFG setting. Our results reveal a fundamental separation between RL for single-agent, MFC, and MFG from the sample efficiency perspective.

READ FULL TEXT
research
10/08/2020

Provable Fictitious Play for General Mean-Field Games

We propose a reinforcement learning algorithm for stationary mean-field ...
research
08/24/2022

Oracle-free Reinforcement Learning in Mean-Field Games along a Single Sample Path

We consider online reinforcement learning in Mean-Field Games. In contra...
research
12/29/2022

Policy Mirror Ascent for Efficient and Independent Learning in Mean Field Games

Mean-field games have been used as a theoretical tool to obtain an appro...
research
06/26/2023

On Imitation in Mean-field Games

We explore the problem of imitation learning (IL) in the context of mean...
research
10/09/2019

Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methods

We investigate reinforcement learning for mean field control problems in...
research
06/07/2021

Concave Utility Reinforcement Learning: the Mean-field Game viewpoint

Concave Utility Reinforcement Learning (CURL) extends RL from linear to ...
research
09/09/2020

Reinforcement Learning in Non-Stationary Discrete-Time Linear-Quadratic Mean-Field Games

In this paper, we study large population multi-agent reinforcement learn...

Please sign up or login with your details

Forgot password? Click here to reset