An Adversarially-Learned Turing Test for Dialog Generation Models

04/16/2021
by   Xiang Gao, et al.
11

The design of better automated dialogue evaluation metrics offers the potential of accelerate evaluation research on conversational AI. However, existing trainable dialogue evaluation models are generally restricted to classifiers trained in a purely supervised manner, which suffer a significant risk from adversarial attacking (e.g., a nonsensical response that enjoys a high classification score). To alleviate this risk, we propose an adversarial training approach to learn a robust model, ATT (Adversarial Turing Test), that discriminates machine-generated responses from human-written replies. In contrast to previous perturbation-based methods, our discriminator is trained by iteratively generating unrestricted and diverse adversarial examples using reinforcement learning. The key benefit of this unrestricted adversarial training approach is allowing the discriminator to improve robustness in an iterative attack-defense game. Our discriminator shows high accuracy on strong attackers including DialoGPT and GPT-3.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/23/2017

Adversarial Learning for Neural Dialogue Generation

In this paper, drawing intuition from the Turing test, we propose using ...
research
03/06/2019

GanDef: A GAN based Adversarial Training Defense for Neural Network Classifier

Machine learning models, especially neural network (NN) classifiers, are...
research
02/28/2022

Probing the Robustness of Trained Metrics for Conversational Dialogue Systems

This paper introduces an adversarial method to stress-test trained metri...
research
06/07/2023

PromptAttack: Probing Dialogue State Trackers with Adversarial Prompts

A key component of modern conversational systems is the Dialogue State T...
research
04/22/2019

blessing in disguise: Designing Robust Turing Test by Employing Algorithm Unrobustness

Turing test was originally proposed to examine whether machine's behavio...
research
09/12/2018

Retrieval-Enhanced Adversarial Training for Neural Response Generation

Dialogue systems are usually built on either generation-based or retriev...
research
11/28/2017

End-to-end Adversarial Learning for Generative Conversational Agents

This paper presents a new adversarial learning method for generative con...

Please sign up or login with your details

Forgot password? Click here to reset