Towards measuring fairness in AI: the Casual Conversations dataset

04/06/2021
by   Caner Hazirbas, et al.
0

This paper introduces a novel dataset to help researchers evaluate their computer vision and audio models for accuracy across a diverse set of age, genders, apparent skin tones and ambient lighting conditions. Our dataset is composed of 3,011 subjects and contains over 45,000 videos, with an average of 15 videos per person. The videos were recorded in multiple U.S. states with a diverse set of adults in various age, gender and apparent skin tone groups. A key feature is that each subject agreed to participate for their likenesses to be used. Additionally, our age and gender annotations are provided by the subjects themselves. A group of trained annotators labeled the subjects' apparent skin tone using the Fitzpatrick skin type scale. Moreover, annotations for videos recorded in low ambient lighting are also provided. As an application to measure robustness of predictions across certain attributes, we provide a comprehensive study on the top five winners of the DeepFake Detection Challenge (DFDC). Experimental evaluation shows that the winning models are less performant on some specific groups of people, such as subjects with darker skin tones and thus may not generalize to all people. In addition, we also evaluate the state-of-the-art apparent age and gender classification methods. Our experiments provides a through analysis on these models in terms of fair treatment of people from various backgrounds.

READ FULL TEXT

page 1

page 2

page 4

page 6

page 8

research
03/08/2023

The Casual Conversations v2 Dataset

This paper introduces a new large consent-driven dataset aimed at assist...
research
10/19/2019

The Deepfake Detection Challenge (DFDC) Preview Dataset

In this paper, we introduce a preview of the Deepfakes Detection Challen...
research
03/30/2022

Automatic Facial Skin Feature Detection for Everyone

Automatic assessment and understanding of facial skin condition have sev...
research
03/10/2021

Understanding the Representation and Representativeness of Age in AI Data Sets

A diverse representation of different demographic groups in AI training ...
research
05/16/2023

Consensus and Subjectivity of Skin Tone Annotation for ML Fairness

Recent advances in computer vision fairness have relied on datasets augm...
research
08/09/2023

FaceSkin: A Privacy Preserving Facial skin patch Dataset for multi Attributes classification

Human facial skin images contain abundant textural information that can ...
research
11/14/2017

Evaluating gender portrayal in Bangladeshi TV

Computer Vision and machine learning methods were previously used to rev...

Please sign up or login with your details

Forgot password? Click here to reset