ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AI

09/20/2021
by   Amanda Cercas Curry, et al.
0

We present the first English corpus study on abusive language towards three conversational AI systems gathered "in the wild": an open-domain social bot, a rule-based chatbot, and a task-based system. To account for the complexity of the task, we take a more `nuanced' approach where our ConvAI dataset reflects fine-grained notions of abuse, as well as views from multiple expert annotators. We find that the distribution of abuse is vastly different compared to other commonly used datasets, with more sexually tinted aggression towards the virtual persona of these systems. Finally, we report results from bench-marking existing models against this data. Unsurprisingly, we find that there is substantial room for improvement with F1 scores below 90

READ FULL TEXT
research
05/02/2022

State-of-the-art in Open-domain Conversational AI: A Survey

We survey SoTA open-domain conversational AI models with the purpose of ...
research
08/30/2018

Pronoun Translation in English-French Machine Translation: An Analysis of Error Types

Pronouns are a long-standing challenge in machine translation. We presen...
research
10/15/2021

Training Conversational Agents with Generative Conversational Networks

Rich, open-domain textual data available on the web resulted in great ad...
research
09/21/2018

Neural Approaches to Conversational AI

The present paper surveys neural approaches to conversational AI that ha...
research
10/07/2018

Conversational Group Detection With Deep Convolutional Networks

Detection of interacting and conversational groups from images has appli...
research
03/17/2022

Conversational Recommendation: A Grand AI Challenge

Animated avatars, which look and talk like humans, are iconic visions of...
research
10/02/2022

Risk-graded Safety for Handling Medical Queries in Conversational AI

Conversational AI systems can engage in unsafe behaviour when handling u...

Please sign up or login with your details

Forgot password? Click here to reset