Learning Parametric Closed-Loop Policies for Markov Potential Games

02/03/2018
by   Sergio Valcarcel Macua, et al.
0

Multiagent systems where the agents interact among themselves and with an stochastic environment can be formalized as stochastic games. We study a subclass, named Markov potential games (MPGs), that appear often in economic and engineering applications when the agents share some common resource. We consider MPGs with continuous state-action variables, coupled constraints and nonconvex rewards. Previous analysis are only valid for very simple cases (convex rewards, invertible dynamics, and no coupled constraints); or considered deterministic dynamics and provided open-loop (OL) analysis, studying strategies that consist in predefined action sequences. We present a closed-loop (CL) analysis for MPGs and consider parametric policies that depend on the current state and where agents adapt to stochastic transitions. We provide verifiable, sufficient and necessary conditions for a stochastic game to be an MPG, even for complex parametric functions (e.g., deep neural networks); and show that a CL Nash equilibrium (NE) can be found (or at least approximated) by solving a related optimal control problem (OCP). This is useful since solving an OCP---a single-objective problem---is usually much simpler than solving the original set of coupled OCPs that form the game---a multiobjective control problem. This is a considerable improvement over previously standard approach. We illustrate the theoretical contributions with an example by applying our approach to a noncooperative communications engineering game. We then solve the game with a deep reinforcement learning algorithm that learns policies that closely approximates an exact variational NE of the game.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

09/14/2020

Multi-Agent Reinforcement Learning in Cournot Games

In this work, we study the interaction of strategic agents in continuous...
10/03/2021

Maximum-Entropy Multi-Agent Dynamic Games: Forward and Inverse Solutions

In this paper, we study the problem of multiple stochastic agents intera...
03/22/2019

Deep Fictitious Play for Stochastic Differential Games

In this paper, we apply the idea of fictitious play to design deep neura...
03/13/2018

Decentralised Learning in Systems with Many, Many Strategic Agents

Although multi-agent reinforcement learning can tackle systems of strate...
06/14/2017

A distributed algorithm for average aggregative games with coupling constraints

We consider the framework of average aggregative games, where the cost f...
09/14/2019

Acquisition Games with Partial-Asymmetric Information

We consider an example of stochastic games with partial, asymmetric and ...
02/24/2020

Optimal strategies in the Fighting Fantasy gaming system: influencing stochastic dynamics by gambling with limited resource

Fighting Fantasy is a popular recreational fantasy gaming system worldwi...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.