REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research

05/05/2022
by   Jessie J. Smith, et al.
0

Transparency around limitations can improve the scientific rigor of research, help ensure appropriate interpretation of research findings, and make research claims more credible. Despite these benefits, the machine learning (ML) research community lacks well-developed norms around disclosing and discussing limitations. To address this gap, we conduct an iterative design process with 30 ML and ML-adjacent researchers to develop and test REAL ML, a set of guided activities to help ML researchers recognize, explore, and articulate the limitations of their research. Using a three-stage interview and survey study, we identify ML researchers' perceptions of limitations, as well as the challenges they face when recognizing, exploring, and articulating limitations. We develop REAL ML to address some of these practical challenges, and highlight additional cultural challenges that will require broader shifts in community norms to address. We hope our study and REAL ML help move the ML research community toward more active and appropriate engagement with limitations.

READ FULL TEXT
research
10/26/2021

Exploring the Role of Machine Learning in Scientific Workflows: Opportunities and Challenges

In this survey, we discuss the challenges of executing scientific workfl...
research
09/06/2022

Use and Misuse of Machine Learning in Anthropology

Machine learning (ML), being now widely accessible to the research commu...
research
01/22/2021

On managing vulnerabilities in AI/ML systems

This paper explores how the current paradigm of vulnerability management...
research
05/27/2022

Looking at Creative ML Blindspots with a Sociological Lens

How can researchers from the creative ML/AI community and sociology of c...
research
02/21/2023

AutoML in The Wild: Obstacles, Workarounds, and Expectations

Automated machine learning (AutoML) is envisioned to make ML techniques ...
research
09/05/2023

Experience and Prediction: A Metric of Hardness for a Novel Litmus Test

In the last decade, the Winograd Schema Challenge (WSC) has become a cen...
research
08/10/2023

Are We Closing the Loop Yet? Gaps in the Generalizability of VIS4ML Research

Visualization for machine learning (VIS4ML) research aims to help expert...

Please sign up or login with your details

Forgot password? Click here to reset