Log In Sign Up

Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure

by   Ben Hutchinson, et al.

Rising concern for the societal implications of artificial intelligence systems has inspired demands for greater transparency and accountability. However the datasets which empower machine learning are often used, shared and re-used with little visibility into the processes of deliberation which led to their creation. Which stakeholder groups had their perspectives included when the dataset was conceived? Which domain experts were consulted regarding how to model subgroups and other phenomena? How were questions of representational biases measured and addressed? Who labeled the data? In this paper, we introduce a rigorous framework for dataset development transparency which supports decision-making and accountability. The framework uses the cyclical, infrastructural and engineering nature of dataset development to draw on best practices from the software development lifecycle. Each stage of the data development lifecycle yields a set of documents that facilitate improved communication and decision-making, as well as drawing attention the value and necessity of careful data work. The proposed framework is intended to contribute to closing the accountability gap in artificial intelligence systems, by making visible the often overlooked work that goes into dataset creation.


page 1

page 2

page 3

page 4


Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing

Rising concern for the societal implications of artificial intelligence ...

Impact Factors and Best Practices to Improve Effort Estimation Strategies and Practices in DevOps

Effort estimation plays an important role in the software development pr...

Agility in Software 2.0 – Notebook Interfaces and MLOps with Buttresses and Rebars

Artificial intelligence through machine learning is increasingly used in...

The Dataset Nutrition Label (2nd Gen): Leveraging Context to Mitigate Harms in Artificial Intelligence

As the production of and reliance on datasets to produce automated decis...

Awareness and use of quantitative decision-making methods in pharmaceutical development

The pharmaceutical industry has experienced increasing costs and sustain...

Trusted Artificial Intelligence: Towards Certification of Machine Learning Applications

Artificial Intelligence is one of the fastest growing technologies of th...

Outlining Traceability: A Principle for Operationalizing Accountability in Computing Systems

Accountability is widely understood as a goal for well governed computer...