Coordinating Fully-Cooperative Agents Using Hierarchical Learning Anticipation

by   Ariyan Bighashdel, et al.

Learning anticipation is a reasoning paradigm in multi-agent reinforcement learning, where agents, during learning, consider the anticipated learning of other agents. There has been substantial research into the role of learning anticipation in improving cooperation among self-interested agents in general-sum games. Two primary examples are Learning with Opponent-Learning Awareness (LOLA), which anticipates and shapes the opponent's learning process to ensure cooperation among self-interested agents in various games such as iterated prisoner's dilemma, and Look-Ahead (LA), which uses learning anticipation to guarantee convergence in games with cyclic behaviors. So far, the effectiveness of applying learning anticipation to fully-cooperative games has not been explored. In this study, we aim to research the influence of learning anticipation on coordination among common-interested agents. We first illustrate that both LOLA and LA, when applied to fully-cooperative games, degrade coordination among agents, causing worst-case outcomes. Subsequently, to overcome this miscoordination behavior, we propose Hierarchical Learning Anticipation (HLA), where agents anticipate the learning of other agents in a hierarchical fashion. Specifically, HLA assigns agents to several hierarchy levels to properly regulate their reasonings. Our theoretical and empirical findings confirm that HLA can significantly improve coordination among common-interested agents in fully-cooperative normal-form games. With HLA, to the best of our knowledge, we are the first to unlock the benefits of learning anticipation for fully-cooperative games.


page 1

page 2

page 3

page 4


Learning Generalizable Risk-Sensitive Policies to Coordinate in Decentralized Multi-Agent General-Sum Games

While various multi-agent reinforcement learning methods have been propo...

HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism

Multi-agent reinforcement learning often suffers from the exponentially ...

Evolving Dyadic Strategies for a Cooperative Physical Task

Many cooperative physical tasks require that individuals play specialize...

Algorithmics of Egalitarian versus Equitable Sequences of Committees

We study the election of sequences of committees, where in each of τ lev...

R2-B2: Recursive Reasoning-Based Bayesian Optimization for No-Regret Learning in Games

This paper presents a recursive reasoning formalism of Bayesian optimiza...

Tackling Asymmetric and Circular Sequential Social Dilemmas with Reinforcement Learning and Graph-based Tit-for-Tat

In many societal and industrial interactions, participants generally pre...

Cooperative Games with Bounded Dependency Degree

Cooperative games provide a framework to study cooperation among self-in...

Please sign up or login with your details

Forgot password? Click here to reset