Reliability Testing for Natural Language Processing Systems

05/06/2021
by   Samson Tan, et al.
20

Questions of fairness, robustness, and transparency are paramount to address before deploying NLP systems. Central to these concerns is the question of reliability: Can NLP systems reliably treat different demographics fairly and function correctly in diverse and noisy environments? To address this, we argue for the need for reliability testing and contextualize it among existing work on improving accountability. We show how adversarial attacks can be reframed for this goal, via a framework for developing reliability tests. We argue that reliability testing – with an emphasis on interdisciplinary collaboration – will enable rigorous and targeted testing, and aid in the enactment and enforcement of industry standards.

READ FULL TEXT
research
10/12/2020

From Hero to Zéroe: A Benchmark of Low-Level Adversarial Attacks

Adversarial attacks are label-preserving modifications to inputs of mach...
research
07/21/2023

Who should I Collaborate with? A Comparative Study of Academia and Industry Research Collaboration in NLP

The goal of our research was to investigate the effects of collaboration...
research
06/14/2019

Principled Frameworks for Evaluating Ethics in NLP Systems

We critique recent work on ethics in natural language processing. Those ...
research
01/03/2022

Robust Natural Language Processing: Recent Advances, Challenges, and Future Directions

Recent natural language processing (NLP) techniques have accomplished hi...
research
05/12/2021

Designing Multimodal Datasets for NLP Challenges

In this paper, we argue that the design and development of multimodal da...
research
08/02/2023

Manual Tests Do Smell! Cataloging and Identifying Natural Language Test Smells

Background: Test smells indicate potential problems in the design and im...
research
03/24/2022

k-Rater Reliability: The Correct Unit of Reliability for Aggregated Human Annotations

Since the inception of crowdsourcing, aggregation has been a common stra...

Please sign up or login with your details

Forgot password? Click here to reset