Likelihood Ratios and Generative Classifiers for Unsupervised Out-of-Domain Detection In Task Oriented Dialog

12/30/2019
by   Varun Gangal, et al.
0

The task of identifying out-of-domain (OOD) input examples directly at test-time has seen renewed interest recently due to increased real world deployment of models. In this work, we focus on OOD detection for natural language sentence inputs to task-based dialog systems. Our findings are three-fold: First, we curate and release ROSTD (Real Out-of-Domain Sentences From Task-oriented Dialog) - a dataset of 4K OOD examples for the publicly available dataset from (Schuster et al. 2019). In contrast to existing settings which synthesize OOD examples by holding out a subset of classes, our examples were authored by annotators with apriori instructions to be out-of-domain with respect to the sentences in an existing dataset. Second, we explore likelihood ratio based approaches as an alternative to currently prevalent paradigms. Specifically, we reformulate and apply these approaches to natural language inputs. We find that they match or outperform the latter on all datasets, with larger improvements on non-artificial OOD benchmarks such as our dataset. Our ablations validate that specifically using likelihood ratios rather than plain likelihood is necessary to discriminate well between OOD and in-domain data. Third, we propose learning a generative classifier and computing a marginal likelihood (ratio) for OOD detection. This allows us to use a principled likelihood while at the same time exploiting training-time labels. We find that this approach outperforms both simple likelihood (ratio) based and other prior approaches. We are hitherto the first to investigate the use of generative classifiers for OOD detection at test-time.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/26/2023

DKAF: KB Arbitration for Learning Task-Oriented Dialog Systems with Dialog-KB Inconsistencies

Task-oriented dialog (TOD) agents often ground their responses on extern...
research
06/07/2019

Likelihood Ratios for Out-of-Distribution Detection

Discriminative neural networks offer little or no performance guarantees...
research
06/17/2022

CookDial: A dataset for task-oriented dialogs grounded in procedural documents

This work presents a new dialog dataset, CookDial, that facilitates rese...
research
01/20/2021

Zero-shot Generalization in Dialog State Tracking through Generative Question Answering

Dialog State Tracking (DST), an integral part of modern dialog systems, ...
research
02/27/2020

Few-shot Natural Language Generation for Task-Oriented Dialog

As a crucial component in task-oriented dialog systems, the Natural Lang...
research
11/30/2022

Reinforced Language Modeling for End-to-End Task Oriented Dialog

In task-oriented dialogs such as MultiWoZ (Budzianowski et al., 2018), a...
research
10/09/2012

Cost-Sensitive Tree of Classifiers

Recently, machine learning algorithms have successfully entered large-sc...

Please sign up or login with your details

Forgot password? Click here to reset