Bi-level Actor-Critic for Multi-agent Coordination

by   Haifeng Zhang, et al.

Coordination is one of the essential problems in multi-agent systems. Typically multi-agent reinforcement learning (MARL) methods treat agents equally and the goal is to solve the Markov game to an arbitrary Nash equilibrium (NE) when multiple equilibra exist, thus lacking a solution for NE selection. In this paper, we treat agents unequally and consider Stackelberg equilibrium as a potentially better convergence point than Nash equilibrium in terms of Pareto superiority, especially in cooperative environments. Under Markov games, we formally define the bi-level reinforcement learning problem in finding Stackelberg equilibrium. We propose a novel bi-level actor-critic learning method that allows agents to have different knowledge base (thus intelligent), while their actions still can be executed simultaneously and distributedly. The convergence proof is given, while the resulting learning algorithm is tested against the state of the arts. We found that the proposed bi-level actor-critic algorithm successfully converged to the Stackelberg equilibria in matrix games and find a asymmetric solution in a highway merge environment.



There are no comments yet.


page 6

page 7


Actor-Critic Provably Finds Nash Equilibria of Linear-Quadratic Mean-Field Games

We study discrete-time mean-field Markov games with infinite numbers of ...

Multi-Agent Generalized Recursive Reasoning

We propose a new reasoning protocol called generalized recursive reasoni...

Resolving Implicit Coordination in Multi-Agent Deep Reinforcement Learning with Deep Q-Networks Game Theory

We address two major challenges of implicit coordination in multi-agent ...

Multi-agent Inverse Reinforcement Learning for General-sum Stochastic Games

This paper addresses the problem of multi-agent inverse reinforcement le...

Optimal Bi-level Lottery Design for Multi-agent Systems

We consider a bi-level lottery where a social planner at the high level ...

Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning

Humans are capable of attributing latent mental contents such as beliefs...

Efficient Cooperation Strategy Generation in Multi-Agent Video Games via Hypergraph Neural Network

The performance of deep reinforcement learning (DRL) in single-agent vid...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.