K-SHAP: Policy Clustering Algorithm for Anonymous State-Action Pairs

02/23/2023
by   Andrea Coletta, et al.
0

Learning agent behaviors from observational data has shown to improve our understanding of their decision-making processes, advancing our ability to explain their interactions with the environment and other agents. While multiple learning techniques have been proposed in the literature, there is one particular setting that has not been explored yet: multi agent systems where agent identities remain anonymous. For instance, in financial markets labeled data that identifies market participant strategies is typically proprietary, and only the anonymous state-action pairs that result from the interaction of multiple market participants are publicly available. As a result, sequences of agent actions are not observable, restricting the applicability of existing work. In this paper, we propose a Policy Clustering algorithm, called K-SHAP, that learns to group anonymous state-action pairs according to the agent policies. We frame the problem as an Imitation Learning (IL) task, and we learn a world-policy able to mimic all the agent behaviors upon different environmental states. We leverage the world-policy to explain each anonymous observation through an additive feature attribution method called SHAP (SHapley Additive exPlanations). Finally, by clustering the explanations we show that we are able to identify different agent policies and group observations accordingly. We evaluate our approach on simulated synthetic market data and a real-world financial dataset. We show that our proposal significantly and consistently outperforms the existing methods, identifying different agent strategies.

READ FULL TEXT

page 2

page 6

page 8

page 15

research
10/25/2021

Towards Realistic Market Simulations: a Generative Adversarial Networks Approach

Simulated environments are increasingly used by trading firms and invest...
research
06/10/2021

Informative Policy Representations in Multi-Agent Reinforcement Learning via Joint-Action Distributions

In multi-agent reinforcement learning, the inherent non-stationarity of ...
research
07/10/2021

Multi-Agent Imitation Learning with Copulas

Multi-agent imitation learning aims to train multiple agents to perform ...
research
03/04/2019

Modeling Social Group Communication with Multi-Agent Imitation Learning

In crowded social scenarios with a myriad of external stimuli, human bra...
research
03/04/2023

Adaptive Predictive Portfolio Management Agent

The paper presents an advanced version of an adaptive market-making agen...
research
08/04/2020

An Imitation from Observation Approach to Sim-to-Real Transfer

The sim to real transfer problem deals with leveraging large amounts of ...

Please sign up or login with your details

Forgot password? Click here to reset