A Tailored Pre-Training Model for Task-Oriented Dialog Generation

by   Jing Gu, et al.
University of California-Davis

The recent success of large pre-trained language models such as BERT and GPT-2 has suggested the effectiveness of incorporating language priors in downstream dialog generation tasks. However, the performance of pre-trained models on the dialog task is not as optimal as expected. In this paper, we propose a Pre-trained Role Alternating Language model (PRAL), designed specifically for task-oriented conversational systems. We adopted (Wu et al., 2019) that models two speakers separately. We also design several techniques, such as start position randomization, knowledge distillation, and history discount to improve pre-training performance. We introduce a task-oriented dialog pretraining dataset by cleaning 13 existing data sets. We test PRAL on three different downstream tasks. The results show that PRAL performs better or on par with state-of-the-art methods.


page 1

page 2

page 3

page 4


Alternating Recurrent Dialog Model with Large-scale Pre-trained Language Models

Existing dialog system models require extensive human annotations and ar...

Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systems

As the labeling cost for different modules in task-oriented dialog (ToD)...

GODEL: Large-Scale Pre-Training for Goal-Directed Dialog

We introduce GODEL (Grounded Open Dialogue Language Model), a large pre-...

UniPCM: Universal Pre-trained Conversation Model with Task-aware Automatic Prompt

Recent research has shown that multi-task pre-training greatly improves ...

SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generation

Recently, pre-training methods have shown remarkable success in task-ori...

Contextual Dynamic Prompting for Response Generation in Task-oriented Dialog Systems

Response generation is one of the critical components in task-oriented d...

RADDLE: An Evaluation Benchmark and Analysis Platform for Robust Task-oriented Dialog Systems

For task-oriented dialog systems to be maximally useful, it must be able...

Please sign up or login with your details

Forgot password? Click here to reset