Opponent Learning Awareness and Modelling in Multi-Objective Normal Form Games

11/14/2020
by   Roxana Rădulescu, et al.
2

Many real-world multi-agent interactions consider multiple distinct criteria, i.e. the payoffs are multi-objective in nature. However, the same multi-objective payoff vector may lead to different utilities for each participant. Therefore, it is essential for an agent to learn about the behaviour of other agents in the system. In this work, we present the first study of the effects of such opponent modelling on multi-objective multi-agent interactions with non-linear utilities. Specifically, we consider two-player multi-objective normal form games with non-linear utility functions under the scalarised expected returns optimisation criterion. We contribute novel actor-critic and policy gradient formulations to allow reinforcement learning of mixed strategies in this setting, along with extensions that incorporate opponent policy reconstruction and learning with opponent learning awareness (i.e., learning while considering the impact of one's policy when anticipating the opponent's learning step). Empirical results in five different MONFGs demonstrate that opponent learning awareness and modelling can drastically alter the learning dynamics in this setting. When equilibria are present, opponent modelling can confer significant benefits on agents that implement it. When there are no Nash equilibria, opponent learning awareness and modelling allows agents to still converge to meaningful solutions that approximate equilibria.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 14

page 15

page 16

01/17/2020

A utility-based analysis of equilibria in multi-objective normal form games

In multi-objective multi-agent systems (MOMAS), agents explicitly consid...
11/17/2021

Preference Communication in Multi-Objective Normal-Form Games

We study the problem of multiple agents learning concurrently in a multi...
12/02/2020

Multi-Objective Optimization of the Textile Manufacturing Process Using Deep-Q-Network Based Multi-Agent Reinforcement Learning

Multi-objective optimization of the textile manufacturing process is an ...
09/06/2019

Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey

The majority of multi-agent system (MAS) implementations aim to optimise...
09/13/2017

Learning with Opponent-Learning Awareness

Multi-agent settings are quickly gathering importance in machine learnin...
01/08/2014

Actor-Critic Algorithms for Learning Nash Equilibria in N-player General-Sum Games

We consider the problem of finding stationary Nash equilibria (NE) in a ...
03/08/2022

COLA: Consistent Learning with Opponent-Learning Awareness

Learning in general-sum games can be unstable and often leads to sociall...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.