Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detection

01/11/2021
by   Alexander Podolskiy, et al.
0

Real-life applications, heavily relying on machine learning, such as dialog systems, demand out-of-domain detection methods. Intent classification models should be equipped with a mechanism to distinguish seen intents from unseen ones so that the dialog agent is capable of rejecting the latter and avoiding undesired behavior. However, despite increasing attention paid to the task, the best practices for out-of-domain intent detection have not yet been fully established. This paper conducts a thorough comparison of out-of-domain intent detection methods. We prioritize the methods, not requiring access to out-of-domain data during training, gathering of which is extremely time- and labor-consuming due to lexical and stylistic variation of user utterances. We evaluate multiple contextual encoders and methods, proven to be efficient, on three standard datasets for intent classification, expanded with out-of-domain utterances. Our main findings show that fine-tuning Transformer-based encoders on in-domain data leads to superior results. Mahalanobis distance, together with utterance representations, derived from Transformer-based encoders, outperforms other methods by a wide margin and establishes new state-of-the-art results for all datasets. The broader analysis shows that the reason for success lies in the fact that the fine-tuned Transformer is capable of constructing homogeneous representations of in-domain utterances, revealing geometrical disparity to out of domain utterances. In turn, the Mahalanobis distance captures this disparity easily.

READ FULL TEXT

page 5

page 6

research
07/27/2021

Energy-based Unknown Intent Detection with Data Manipulation

Unknown intent detection aims to identify the out-of-distribution (OOD) ...
research
02/04/2021

Generalized Zero-shot Intent Detection via Commonsense Knowledge

Identifying user intents from natural language utterances is a crucial s...
research
10/17/2020

Example-Driven Intent Prediction with Observers

A key challenge of dialog systems research is to effectively and efficie...
research
08/16/2021

A Single Example Can Improve Zero-Shot Data Generation

Sub-tasks of intent classification, such as robustness to distribution s...
research
09/04/2021

Error Detection in Large-Scale Natural Language Understanding Systems Using Transformer Models

Large-scale conversational assistants like Alexa, Siri, Cortana and Goog...
research
12/07/2018

Intent Detection for code-mix utterances in task oriented dialogue systems

Intent detection is an essential component of task oriented dialogue sys...
research
05/09/2023

Going beyond research datasets: Novel intent discovery in the industry setting

Novel intent discovery automates the process of grouping similar message...

Please sign up or login with your details

Forgot password? Click here to reset