Evaluating a Methodology for Increasing AI Transparency: A Case Study

01/24/2022
by   David Piorkowski, et al.
0

In reaction to growing concerns about the potential harms of artificial intelligence (AI), societies have begun to demand more transparency about how AI models and systems are created and used. To address these concerns, several efforts have proposed documentation templates containing questions to be answered by model developers. These templates provide a useful starting point, but no single template can cover the needs of diverse documentation consumers. It is possible in principle, however, to create a repeatable methodology to generate truly useful documentation. Richards et al. [25] proposed such a methodology for identifying specific documentation needs and creating templates to address those needs. Although this is a promising proposal, it has not been evaluated. This paper presents the first evaluation of this user-centered methodology in practice, reporting on the experiences of a team in the domain of AI for healthcare that adopted it to increase transparency for several AI models. The methodology was found to be usable by developers not trained in user-centered techniques, guiding them to creating a documentation template that addressed the specific needs of their consumers while still being reusable across different models and use cases. Analysis of the benefits and costs of this methodology are reviewed and suggestions for further improvement in both the methodology and supporting tools are summarized.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/21/2023

Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience

Despite the widespread use of artificial intelligence (AI), designing us...
research
06/24/2020

A Methodology for Creating AI FactSheets

As AI models and services are used in a growing number of highstakes are...
research
11/11/2019

Experiences with Improving the Transparency of AI Models and Services

AI models and services are used in a growing number of highstakes areas,...
research
06/23/2023

Use case cards: a use case reporting framework inspired by the European AI Act

Despite recent efforts by the Artificial Intelligence (AI) community to ...
research
03/09/2023

Users are the North Star for AI Transparency

Despite widespread calls for transparent artificial intelligence systems...
research
06/02/2023

AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap

The rise of powerful large language models (LLMs) brings about tremendou...
research
01/23/2021

Outlining Traceability: A Principle for Operationalizing Accountability in Computing Systems

Accountability is widely understood as a goal for well governed computer...

Please sign up or login with your details

Forgot password? Click here to reset