Auditing Cross-Cultural Consistency of Human-Annotated Labels for Recommendation Systems

by   Rock Yuren Pang, et al.

Recommendation systems increasingly depend on massive human-labeled datasets; however, the human annotators hired to generate these labels increasingly come from homogeneous backgrounds. This poses an issue when downstream predictive models – based on these labels – are applied globally to a heterogeneous set of users. We study this disconnect with respect to the labels themselves, asking whether they are “consistently conceptualized” across annotators of different demographics. In a case study of video game labels, we conduct a survey on 5,174 gamers, identify a subset of inconsistently conceptualized game labels, perform causal analyses, and suggest both cultural and linguistic reasons for cross-country differences in label annotation. We further demonstrate that predictive models of game annotations perform better on global train sets as opposed to homogeneous (single-country) train sets. Finally, we provide a generalizable framework for practitioners to audit their own data annotation processes for consistent label conceptualization, and encourage practitioners to consider global inclusivity in recommendation systems starting from the early stages of annotator recruitment and data-labeling.


page 19

page 20

page 21


The Importance of Socio-Cultural Differences for Annotating and Detecting the Affective States of Students

The development of real-time affect detection models often depends upon ...

Label quality in AffectNet: results of crowd-based re-annotation

AffectNet is one of the most popular resources for facial expression rec...

The Origin and Value of Disagreement Among Data Labelers: A Case Study of the Individual Difference in Hate Speech Annotation

Human annotated data is the cornerstone of today's artificial intelligen...

On the Ramifications of Human Label Uncertainty

Humans exhibit disagreement during data labeling. We term this disagreem...

Learning with Different Amounts of Annotation: From Zero to Many Labels

Training NLP systems typically assumes access to annotated data that has...

Chord Label Personalization through Deep Learning of Integrated Harmonic Interval-based Representations

The increasing accuracy of automatic chord estimation systems, the avail...

Please sign up or login with your details

Forgot password? Click here to reset