DeepAI AI Chat
Log In Sign Up

Learning to Advise Humans By Leveraging Algorithm Discretion

by   Nicholas Wolczynski, et al.
The University of Texas at Austin
The University of Iowa

Expert decision-makers (DMs) in high-stakes AI-advised (AIDeT) settings receive and reconcile recommendations from AI systems before making their final decisions. We identify distinct properties of these settings which are key to developing AIDeT models that effectively benefit team performance. First, DMs in AIDeT settings exhibit algorithm discretion behavior (ADB), i.e., an idiosyncratic tendency to imperfectly accept or reject algorithmic recommendations for any given decision task. Second, DMs incur contradiction costs from exerting decision-making resources (e.g., time and effort) when reconciling AI recommendations that contradict their own judgment. Third, the human's imperfect discretion and reconciliation costs introduce the need for the AI to offer advice selectively. We refer to the task of developing AI to advise humans in AIDeT settings as learning to advise and we address this task by first introducing the AIDeT-Learning Framework. Additionally, we argue that leveraging the human partner's ADB is key to maximizing the AIDeT's decision accuracy while regularizing for contradiction costs. Finally, we instantiate our framework to develop TeamRules (TR): an algorithm that produces rule-based models and recommendations for AIDeT settings. TR is optimized to selectively advise a human and to trade-off contradiction costs and team accuracy for a given environment by leveraging the human partner's ADB. Evaluations on synthetic and real-world benchmark datasets with a variety of simulated human accuracy and discretion behaviors show that TR robustly improves the team's objective across settings over interpretable, rule-based alternatives.


page 1

page 2

page 3

page 4


Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance

Increasingly, organizations are pairing humans with AI systems to improv...

Optimizing AI for Teamwork

In many high-stakes domains such as criminal justice, finance, and healt...

Modeling Human-AI Team Decision Making

AI and humans bring complementary skills to group deliberations. Modelin...

Role of Human-AI Interaction in Selective Prediction

Recent work has shown the potential benefit of selective prediction syst...

Human-AI Collaboration with Bandit Feedback

Human-machine complementarity is important when neither the algorithm no...

An Interpretable Algorithm for Uveal Melanoma Subtyping from Whole Slide Cytology Images

Algorithmic decision support is rapidly becoming a staple of personalize...

Learning Complementary Policies for Human-AI Teams

Human-AI complementarity is important when neither the algorithm nor the...