Toward a Perspectivist Turn in Ground Truthing for Predictive Computing

09/09/2021
by   Valerio Basile, et al.
0

Most Artificial Intelligence applications are based on supervised machine learning (ML), which ultimately grounds on manually annotated data. The annotation process is often performed in terms of a majority vote and this has been proved to be often problematic, as highlighted by recent studies on the evaluation of ML models. In this article we describe and advocate for a different paradigm, which we call data perspectivism, which moves away from traditional gold standard datasets, towards the adoption of methods that integrate the opinions and perspectives of the human subjects involved in the knowledge representation step of ML processes. Drawing on previous works which inspired our proposal we describe the potential of our proposal for not only the more subjective tasks (e.g. those related to human language) but also to tasks commonly understood as objective (e.g. medical decision making), and present the main advantages of adopting a perspectivist stance in ML, as well as possible disadvantages, and various ways in which such a stance can be implemented in practice. Finally, we share a set of recommendations and outline a research agenda to advance the perspectivist stance in ML.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/06/2019

Unfairness towards subjective opinions in Machine Learning

Despite the high interest for Machine Learning (ML) in academia and indu...
research
02/18/2023

Machine Love

While ML generates much economic value, many of us have problematic rela...
research
06/20/2022

The Role of Machine Learning in Cybersecurity

Machine Learning (ML) represents a pivotal technology for current and fu...
research
09/01/2022

Making Intelligence: Ethics, IQ, and ML Benchmarks

The ML community recognizes the importance of anticipating and mitigatin...
research
06/08/2022

Resolving the Human Subjects Status of Machine Learning's Crowdworkers

In recent years, machine learning (ML) has come to rely more heavily on ...
research
03/02/2023

Artificial Intelligence for Dementia Research Methods Optimization

Introduction: Machine learning (ML) has been extremely successful in ide...
research
08/23/2021

Analyzing the Granularity and Cost of Annotation in Clinical Sequence Labeling

Well-annotated datasets, as shown in recent top studies, are becoming mo...

Please sign up or login with your details

Forgot password? Click here to reset