Natural Actor-Critic Converges Globally for Hierarchical Linear Quadratic Regulator

12/14/2019

∙

Multi-agent reinforcement learning has been successfully applied to a number of challenging problems. Despite these empirical successes, theoretical understanding of different algorithms is lacking, primarily due to the curse of dimensionality caused by the exponential growth of the state-action space with the number of agents. We study a fundamental problem of multi-agent linear quadratic regulator in a setting where the agents are partially exchangeable. In this setting, we develop a hierarchical actor-critic algorithm, whose computational complexity is independent of the total number of agents, and prove its global linear convergence to the optimal policy. As linear quadratic regulators are often used to approximate general dynamic systems, this paper provided an important step towards better understanding of general hierarchical mean-field multi-agent reinforcement learning.

READ FULL TEXT

Natural Actor-Critic Converges Globally for Hierarchical Linear Quadratic Regulator

On the Global Convergence of Actor-Critic: A Case for Linear Quadratic Regulator with Ergodic Cost

Distributed off-Policy Actor-Critic Reinforcement Learning with Policy Consensus

Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approach

Attention Actor-Critic algorithm for Multi-Agent Constrained Co-operative Reinforcement Learning

Q-Learning for Mean-Field Controls

RoMFAC: A Robust Mean-Field Actor-Critic Reinforcement Learning against Adversarial Perturbations on States

Infer Your Enemies and Know Yourself, Learning in Real-Time Bidding with Partially Observable Opponents

Natural Actor-Critic Converges Globally for Hierarchical Linear Quadratic Regulator

Related Research

On the Global Convergence of Actor-Critic: A Case for Linear Quadratic Regulator with Ergodic Cost

Distributed off-Policy Actor-Critic Reinforcement Learning with Policy Consensus

Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approach

Attention Actor-Critic algorithm for Multi-Agent Constrained Co-operative Reinforcement Learning

Q-Learning for Mean-Field Controls

RoMFAC: A Robust Mean-Field Actor-Critic Reinforcement Learning against Adversarial Perturbations on States

Infer Your Enemies and Know Yourself, Learning in Real-Time Bidding with Partially Observable Opponents