DeepAI
Log In Sign Up

DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue

09/28/2020
by   Shikib Mehri, et al.
23

A long-standing goal of task-oriented dialogue research is the ability to flexibly adapt dialogue models to new domains. To progress research in this direction, we introduce DialoGLUE (Dialogue Language Understanding Evaluation), a public benchmark consisting of 7 task-oriented dialogue datasets covering 4 distinct natural language understanding tasks, designed to encourage dialogue research in representation-based transfer, domain adaptation, and sample-efficient task learning. We release several strong baseline models, demonstrating performance improvements over a vanilla BERT architecture and state-of-the-art results on 5 out of 7 tasks, by pre-training on a large open-domain dialogue corpus and task-adaptive self-supervised training. Through the DialoGLUE benchmark, the baseline methods, and our evaluation scripts, we hope to facilitate progress towards the goal of developing more general task-oriented dialogue models.

READ FULL TEXT

page 1

page 2

page 3

page 4

04/15/2020

ToD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogues

The use of pre-trained language models has emerged as a promising direct...
07/22/2019

Why Build an Assistant in Minecraft?

In this document we describe a rationale for a research program aimed at...
05/25/2022

DialogZoo: Large-Scale Dialog-Oriented Task Learning

Building unified conversational agents has been a long-standing goal of ...
07/03/2018

Intent Generation for Goal-Oriented Dialogue Systems based on Schema.org Annotations

Goal-oriented dialogue systems typically communicate with a backend (e.g...
06/12/2020

A Generative Model for Joint Natural Language Understanding and Generation

Natural language understanding (NLU) and natural language generation (NL...
02/26/2021

Evaluate On-the-job Learning Dialogue Systems and a Case Study for Natural Language Understanding

On-the-job learning consists in continuously learning while being used i...
05/20/2022

Robust Task-Oriented Dialogue Generation with Contrastive Pre-training and Adversarial Filtering

Data artifacts incentivize machine learning models to learn non-transfer...