Relational Reasoning via Set Transformers: Provable Efficiency and Applications to MARL

09/20/2022
by   Fengzhuo Zhang, et al.
1

The cooperative Multi-A gent R einforcement Learning (MARL) with permutation invariant agents framework has achieved tremendous empirical successes in real-world applications. Unfortunately, the theoretical understanding of this MARL problem is lacking due to the curse of many agents and the limited exploration of the relational reasoning in existing works. In this paper, we verify that the transformer implements complex relational reasoning, and we propose and analyze model-free and model-based offline MARL algorithms with the transformer approximators. We prove that the suboptimality gaps of the model-free and model-based algorithms are independent of and logarithmic in the number of agents respectively, which mitigates the curse of many agents. These results are consequences of a novel generalization error bound of the transformer and a novel analysis of the Maximum Likelihood Estimate (MLE) of the system dynamics with the transformer. Our model-based algorithm is the first provably efficient MARL algorithm that explicitly exploits the permutation invariance of the agents. Our improved generalization bound may be of independent interest and is applicable to other regression problems related to the transformer beyond MARL.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/08/2021

Model-based versus Model-free Deep Reinforcement Learning for Autonomous Racing Cars

Despite the rich theoretical foundation of model-based deep reinforcemen...
research
01/14/2022

Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learning

Offline reinforcement learning (RL) Algorithms are often designed with e...
research
11/02/2021

Procedural Generalization by Planning with Self-Supervised World Models

One of the key promises of model-based reinforcement learning is the abi...
research
09/22/2020

Is Q-Learning Provably Efficient? An Extended Analysis

This work extends the analysis of the theoretical results presented with...
research
07/12/2020

Relational-Grid-World: A Novel Relational Reasoning Environment and An Agent Model for Relational Information Extraction

Reinforcement learning (RL) agents are often designed specifically for a...
research
03/13/2023

Transformer-based World Models Are Happy With 100k Interactions

Deep neural networks have been successful in many reinforcement learning...
research
06/08/2023

Negotiated Reasoning: On Provably Addressing Relative Over-Generalization

Over-generalization is a thorny issue in cognitive science, where people...

Please sign up or login with your details

Forgot password? Click here to reset