Dialogue-oriented Pre-training

06/01/2021
by   Yi Xu, et al.
0

Pre-trained language models (PrLM) has been shown powerful in enhancing a broad range of downstream tasks including various dialogue related ones. However, PrLMs are usually trained on general plain text with common language model (LM) training objectives, which cannot sufficiently capture dialogue exclusive features due to the limitation of such training setting, so that there is an immediate need to fill the gap between a specific dialogue task and the LM task. As it is unlikely to collect huge dialogue data for dialogue-oriented pre-training, in this paper, we propose three strategies to simulate the conversation features on general plain text. Our proposed method differs from existing post-training methods that it may yield a general-purpose PrLM and does not individualize to any detailed task while keeping the capability of learning dialogue related features including speaker awareness, continuity and consistency. The resulted Dialog-PrLM is fine-tuned on three public multi-turn dialogue datasets and helps achieve significant and consistent improvement over the plain PrLMs.

READ FULL TEXT

page 1

page 2

page 4

page 6

page 7

page 8

page 9

page 10

research
06/17/2023

FutureTOD: Teaching Future Knowledge to Pre-trained Language Model for Task-Oriented Dialogue

Pre-trained language models based on general text enable huge success in...
research
05/24/2022

D4: a Chinese Dialogue Dataset for Depression-Diagnosis-Oriented Chat

In a depression-diagnosis-directed clinical session, doctors initiate a ...
research
05/23/2021

Structural Pre-training for Dialogue Comprehension

Pre-trained language models (PrLMs) have demonstrated superior performan...
research
06/06/2022

Domain-specific Language Pre-training for Dialogue Comprehension on Clinical Inquiry-Answering Conversations

There is growing interest in the automated extraction of relevant inform...
research
05/18/2023

Causal Document-Grounded Dialogue Pre-training

The goal of document-grounded dialogue (DocGD) is to generate a response...
research
10/26/2020

Probing Task-Oriented Dialogue Representation from Language Models

This paper investigates pre-trained language models to find out which mo...
research
09/10/2022

OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialogue

This paper presents an ontology-aware pretrained language model (OPAL) f...

Please sign up or login with your details

Forgot password? Click here to reset