Planning for Natural Language Failures with the AI Playbook

06/18/2021
by   mkhong87, et al.
0

Prototyping AI user experiences is challenging due in part to probabilistic AI models making it difficult to anticipate, test, and mitigate AI failures before deployment. In this work, we set out to support practitioners with early AI prototyping, with a focus on natural language (NL)-based technologies. Our interviews with 12 NL practitioners from a large technology company revealed that, in addition to challenges prototyping AI, prototyping was often not happening at all or focused only on idealized scenarios due to a lack of tools and tight timelines. These findings informed our design of the AI Playbook, an interactive and low-cost tool we developed to encourage proactive and systematic consideration of AI errors before deployment. Our evaluation of the AI Playbook demonstrates its potential to 1) encourage product teams to prioritize both ideal and failure scenarios, 2) standardize the articulation of AI failures from a user experience perspective, and 3) act as a boundary object between user experience designers, data scientists, and engineers.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/22/2023

fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks

To design with AI models, user experience (UX) designers must assess the...
research
08/21/2022

Performance, Opaqueness, Consequences, and Assumptions: Simple questions for responsible planning of machine learning solutions

The data revolution has generated a huge demand for data-driven solution...
research
01/28/2023

Investigating How Practitioners Use Human-AI Guidelines: A Case Study on the People + AI Guidebook

Artificial intelligence (AI) presents new challenges for the user experi...
research
02/09/2023

Zeno: An Interactive Framework for Behavioral Evaluation of Machine Learning

Machine learning models with high accuracy on test data can still produc...
research
03/27/2021

Towards Tool-Support for Interactive-Machine Learning Applications in the Android Ecosystem

Consumer applications are becoming increasingly smarter and most of them...
research
07/04/2022

Human-AI Guidelines in Practice: Leaky Abstractions as an Enabler in Collaborative Software Teams

In conventional software development, user experience (UX) designers and...

Please sign up or login with your details

Forgot password? Click here to reset