TicketTalk: Toward human-level performance with end-to-end, transaction-based dialog systems

12/23/2020
by   Bill Byrne, et al.
0

We present a data-driven, end-to-end approach to transaction-based dialog systems that performs at near-human levels in terms of verbal response quality and factual grounding accuracy. We show that two essential components of the system produce these results: a sufficiently large and diverse, in-domain labeled dataset, and a neural network-based, pre-trained model that generates both verbal responses and API call predictions. In terms of data, we introduce TicketTalk, a movie ticketing dialog dataset with 23,789 annotated conversations. The movie ticketing conversations range from completely open-ended and unrestricted to more structured, both in terms of their knowledge base, discourse features, and number of turns. In qualitative human evaluations, model-generated responses trained on just 10,000 TicketTalk dialogs were rated to "make sense" 86.5 percent of the time, almost the same as human responses in the same contexts. Our simple, API-focused annotation schema results in a much easier labeling task making it faster and more cost effective. It is also the key component for being able to predict API calls accurately. We handle factual grounding by incorporating API calls in the training data, allowing our model to learn which actions to take and when. Trained on the same 10,000-dialog set, the model's API call predictions were rated to be correct 93.9 percent of the time in our evaluations, surpassing the ratings for the corresponding human labels. We show how API prediction and response generation scores improve as the dataset size incrementally increases from 5000 to 21,000 dialogs. Our analysis also clearly illustrates the benefits of pre-training. We are publicly releasing the TicketTalk dataset with this paper to facilitate future work on transaction-based dialogs.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/01/2019

Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset

A significant barrier to progress in data-driven approaches to building ...
research
08/20/2017

An End-to-End Trainable Neural Network Model with Belief Tracking for Task-Oriented Dialog

We present a novel end-to-end trainable neural network model for task-or...
research
10/22/2022

Robots-Dont-Cry: Understanding Falsely Anthropomorphic Utterances in Dialog Systems

Dialog systems are often designed or trained to output human-like respon...
research
04/18/2019

ConvLab: Multi-Domain End-to-End Dialog System Platform

We present ConvLab, an open-source multi-domain end-to-end dialog system...
research
10/05/2020

Effects of Naturalistic Variation in Goal-Oriented Dialog

Existing benchmarks used to evaluate the performance of end-to-end neura...
research
09/15/2020

Dialogue Response Ranking Training with Large-Scale Human Feedback Data

Existing open-domain dialog models are generally trained to minimize the...
research
03/10/2021

FiLiPo: A Sample Driven Approach for Finding Linkage Points between RDF Data and APIs (Extended Version)

Data integration is an important task in order to create comprehensive R...

Please sign up or login with your details

Forgot password? Click here to reset