AI Ethics Statements – Analysis and lessons learnt from NeurIPS Broader Impact Statements

11/02/2021
by   Carolyn Ashurst, et al.
1

Ethics statements have been proposed as a mechanism to increase transparency and promote reflection on the societal impacts of published research. In 2020, the machine learning (ML) conference NeurIPS broke new ground by requiring that all papers include a broader impact statement. This requirement was removed in 2021, in favour of a checklist approach. The 2020 statements therefore provide a unique opportunity to learn from the broader impact experiment: to investigate the benefits and challenges of this and similar governance mechanisms, as well as providing an insight into how ML researchers think about the societal impacts of their own work. Such learning is needed as NeurIPS and other venues continue to question and adapt their policies. To enable this, we have created a dataset containing the impact statements from all NeurIPS 2020 papers, along with additional information such as affiliation type, location and subject area, and a simple visualisation tool for exploration. We also provide an initial quantitative analysis of the dataset, covering representation, engagement, common themes, and willingness to discuss potential harms alongside benefits. We investigate how these vary by geography, affiliation type and subject area. Drawing on these findings, we discuss the potential benefits and negative outcomes of ethics statement requirements, and their possible causes and associated challenges. These lead us to several lessons to be learnt from the 2020 requirement: (i) the importance of creating the right incentives, (ii) the need for clear expectations and guidance, and (iii) the importance of transparency and constructive deliberation. We encourage other researchers to use our dataset to provide additional analysis, to further our understanding of how researchers responded to this requirement, and to investigate the benefits and challenges of this and related mechanisms.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/30/2021

Institutionalising Ethics in AI through Broader Impact Requirements

Turning principles into practice is one of the most pressing challenges ...
research
05/11/2021

Unpacking the Expressed Consequences of AI Research in Broader Impact Statements

The computer science research community and the broader public have beco...
research
11/25/2020

Like a Researcher Stating Broader Impact For the Very First Time

In requiring that a statement of broader impact accompany all submission...
research
07/11/2021

Adversarial for Good? How the Adversarial ML Community's Values Impede Socially Beneficial Uses of Attacks

Attacks from adversarial machine learning (ML) have the potential to be ...
research
07/19/2021

Diversity in Sociotechnical Machine Learning Systems

There has been a surge of recent interest in sociocultural diversity in ...
research
03/16/2022

On the Use of Fine-grained Vulnerable Code Statements for Software Vulnerability Assessment Models

Many studies have developed Machine Learning (ML) approaches to detect S...

Please sign up or login with your details

Forgot password? Click here to reset