DeepAI AI Chat
Log In Sign Up

Dimension-Free Rates for Natural Policy Gradient in Multi-Agent Reinforcement Learning

by   Carlo Alfano, et al.

Cooperative multi-agent reinforcement learning is a decentralized paradigm in sequential decision making where agents distributed over a network iteratively collaborate with neighbors to maximize global (network-wide) notions of rewards. Exact computations typically involve a complexity that scales exponentially with the number of agents. To address this curse of dimensionality, we design a scalable algorithm based on the Natural Policy Gradient framework that uses local information and only requires agents to communicate with neighbors within a certain range. Under standard assumptions on the spatial decay of correlations for the transition dynamics of the underlying Markov process and the localized learning policy, we show that our algorithm converges to the globally optimal policy with a dimension-free statistical and computational complexity, incurring a localization error that does not depend on the number of agents and converges to zero exponentially fast as a function of the range of communication.


page 1

page 2

page 3

page 4


Scalable Multi-Agent Reinforcement Learning with General Utilities

We study the scalable multi-agent reinforcement learning (MARL) with gen...

Global Convergence of Localized Policy Iteration in Networked Multi-Agent Reinforcement Learning

We study a multi-agent reinforcement learning (MARL) problem where the a...

Distributed Policy Gradient with Variance Reduction in Multi-Agent Reinforcement Learning

This paper studies a distributed policy gradient in collaborative multi-...

DARL1N: Distributed multi-Agent Reinforcement Learning with One-hop Neighbors

Most existing multi-agent reinforcement learning (MARL) methods are limi...

A Law of Iterated Logarithm for Multi-Agent Reinforcement Learning

In Multi-Agent Reinforcement Learning (MARL), multiple agents interact w...

Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination

A key challenge for Multiagent RL (Reinforcement Learning) is the design...

Task-Oriented Communication Design at Scale

With countless promising applications in various domains such as IoT and...