Data and its (dis)contents: A survey of dataset development and use in machine learning research

by   Amandalynne Paullada, et al.

Datasets have played a foundational role in the advancement of machine learning research. They form the basis for the models we design and deploy, as well as our primary medium for benchmarking and evaluation. Furthermore, the ways in which we collect, construct and share these datasets inform the kinds of problems the field pursues and the methods explored in algorithm development. However, recent work from a breadth of perspectives has revealed the limitations of predominant practices in dataset collection and use. In this paper, we survey the many concerns raised about the way we collect and use data in machine learning and advocate that a more cautious and thorough understanding of data is necessary to address several of the practical and ethical issues of the field.



There are no comments yet.


page 1

page 2

page 3

page 4


Bringing the People Back In: Contesting Benchmark Machine Learning Datasets

In response to algorithmic unfairness embedded in sociotechnical systems...

Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research

Benchmark datasets play a central role in the organization of machine le...

Challenges in Deploying Machine Learning: a Survey of Case Studies

In recent years, machine learning has received increased interest both a...

Pitfalls in Machine Learning Research: Reexamining the Development Cycle

Machine learning has the potential to fuel further advances in data scie...

HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine Learning

Recent advancements in machine learning research, i.e., deep learning, i...

A Reflection on Learning from Data: Epistemology Issues and Limitations

Although learning from data is effective and has achieved significant mi...

Bad practices in evaluation methodology relevant to class-imbalanced problems

For research to go in the right direction, it is essential to be able to...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.