The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards

05/09/2018
by   Sarah Holland, et al.
0

Artificial intelligence (AI) systems built on incomplete or biased data will often exhibit problematic outcomes. Current methods of data analysis, particularly before model development, are costly and not standardized. The Dataset Nutrition Label (the Label) is a diagnostic framework that lowers the barrier to standardized data analysis by providing a distilled yet comprehensive overview of dataset "ingredients" before AI model development. Building a Label that can be applied across domains and data types requires that the framework itself be flexible and adaptable; as such, the Label is comprised of diverse qualitative and quantitative modules generated through multiple statistical and probabilistic modelling backends, but displayed in a standardized format. To demonstrate and advance this concept, we generated and published an open source prototype with seven sample modules on the ProPublica Dollars for Docs dataset. The benefits of the Label are manyfold. For data specialists, the Label will drive more robust data analysis practices, provide an efficient way to select the best dataset for their purposes, and increase the overall quality of AI models as a result of more robust training datasets and the ability to check for issues at the time of model development. For those building and publishing datasets, the Label creates an expectation of explanation, which will drive better data collection practices. We also explore the limitations of the Label, including the challenges of generalizing across diverse datasets, and the risk of using "ground truth" data as a comparison dataset. We discuss ways to move forward given the limitations identified. Lastly, we lay out future directions for the Dataset Nutrition Label project, including research and public policy agendas to further advance consideration of the concept.

READ FULL TEXT

page 3

page 8

research
05/09/2023

The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation

We present The Vault, an open-source, large-scale code-text dataset desi...
research
11/09/2021

Statistical Perspectives on Reliability of Artificial Intelligence Systems

Artificial intelligence (AI) systems have become increasingly popular in...
research
01/10/2022

The Dataset Nutrition Label (2nd Gen): Leveraging Context to Mitigate Harms in Artificial Intelligence

As the production of and reliance on datasets to produce automated decis...
research
01/07/2020

Artificial Intelligence for Social Good: A Survey

Artificial intelligence for social good (AI4SG) is a research theme that...
research
11/16/2021

Who Decides if AI is Fair? The Labels Problem in Algorithmic Auditing

Labelled "ground truth" datasets are routinely used to evaluate and audi...
research
05/05/2019

Enhanced Labeling of Issue Reports (with F^3T)

Standard automatic methods for recognizing problematic code can be great...

Please sign up or login with your details

Forgot password? Click here to reset