The Utility of Explainable AI in Ad Hoc Human-Machine Teaming

by   Rohan Paleja, et al.

Recent advances in machine learning have led to growing interest in Explainable AI (xAI) to enable humans to gain insight into the decision-making of machine learning models. Despite this recent interest, the utility of xAI techniques has not yet been characterized in human-machine teaming. Importantly, xAI offers the promise of enhancing team situational awareness (SA) and shared mental model development, which are the key characteristics of effective human-machine teams. Rapidly developing such mental models is especially critical in ad hoc human-machine teaming, where agents do not have a priori knowledge of others' decision-making strategies. In this paper, we present two novel human-subject experiments quantifying the benefits of deploying xAI techniques within a human-machine teaming scenario. First, we show that xAI techniques can support SA (p<0.05). Second, we examine how different SA levels induced via a collaborative AI policy abstraction affect ad hoc human-machine teaming performance. Importantly, we find that the benefits of xAI are not universal, as there is a strong dependence on the composition of the human-machine team. Novices benefit from xAI providing increased SA (p<0.05) but are susceptible to cognitive overhead (p<0.05). On the other hand, expert performance degrades with the addition of xAI-based support (p<0.05), indicating that the cost of paying attention to the xAI outweighs the benefits obtained from being provided additional information to enhance SA. Our results demonstrate that researchers must deliberately design and deploy the right xAI techniques in the right scenario by carefully considering human-machine team composition and how the xAI method augments SA.


page 2

page 5

page 16

page 20


Team Behavior in Interactive Dynamic Influence Diagrams with Applications to Ad Hoc Teams

Planning for ad hoc teamwork is challenging because it involves agents c...

Behavioral Differences is the Key of Ad-hoc Team Cooperation in Multiplayer Games Hanabi

Ad-hoc team cooperation is the problem of cooperating with other players...

Knowledge-based and Data-driven Reasoning and Learning for Ad Hoc Teamwork

We present an architecture for ad hoc teamwork, which refers to collabor...

Adapting Behaviour Based On Trust In Human-Agent Ad Hoc Teamwork

This work proposes a framework that incorporates trust in an ad hoc team...

Accounts, Accountability and Agency for Safe and Ethical AI

We examine the problem of explainable AI (xAI) and explore what deliveri...

An Experimentation Platform for Explainable Coalition Situational Understanding

We present an experimentation platform for coalition situational underst...

Enabling Explainable Fusion in Deep Learning with Fuzzy Integral Neural Networks

Information fusion is an essential part of numerous engineering systems ...

Please sign up or login with your details

Forgot password? Click here to reset