Collective eXplainable AI: Explaining Cooperative Strategies and Agent Contribution in Multiagent Reinforcement Learning with Shapley Values

10/04/2021
by   Alexandre Heuillet, et al.
2

While Explainable Artificial Intelligence (XAI) is increasingly expanding more areas of application, little has been applied to make deep Reinforcement Learning (RL) more comprehensible. As RL becomes ubiquitous and used in critical and general public applications, it is essential to develop methods that make it better understood and more interpretable. This study proposes a novel approach to explain cooperative strategies in multiagent RL using Shapley values, a game theory concept used in XAI that successfully explains the rationale behind decisions taken by Machine Learning algorithms. Through testing common assumptions of this technique in two cooperation-centered socially challenging multi-agent environments environments, this article argues that Shapley values are a pertinent way to evaluate the contribution of players in a cooperative multi-agent RL context. To palliate the high overhead of this method, Shapley values are approximated using Monte Carlo sampling. Experimental results on Multiagent Particle and Sequential Social Dilemmas show that Shapley values succeed at estimating the contribution of each agent. These results could have implications that go beyond games in economics, (e.g., for non-discriminatory decision making, ethical and responsible AI-derived decisions or policy making under fairness constraints). They also expose how Shapley values only give general explanations about a model and cannot explain a single run, episode nor justify precise actions taken by agents. Future work should focus on addressing these critical aspects.

READ FULL TEXT

page 1

page 6

page 7

page 8

research
04/26/2022

Toward Policy Explanations for Multi-Agent Reinforcement Learning

Advances in multi-agent reinforcement learning (MARL) enable sequential ...
research
06/09/2023

Explaining Reinforcement Learning with Shapley Values

For reinforcement learning systems to be widely adopted, their users mus...
research
06/11/2023

Herd's Eye View: Improving Game AI Agent Learning with Collaborative Perception

We present a novel perception model named Herd's Eye View (HEV) that ado...
research
08/20/2021

Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey

Broad Explainable Artificial Intelligence moves away from interpreting i...
research
05/06/2023

Explaining RL Decisions with Trajectories

Explanation is a key component for the adoption of reinforcement learnin...
research
11/20/2022

Revealing Robust Oil and Gas Company Macro-Strategies using Deep Multi-Agent Reinforcement Learning

The energy transition potentially poses an existential risk for major in...
research
05/17/2023

Pittsburgh Learning Classifier Systems for Explainable Reinforcement Learning: Comparing with XCS

Interest in reinforcement learning (RL) has recently surged due to the a...

Please sign up or login with your details

Forgot password? Click here to reset