Hierarchical Modeling for Out-of-Scope Domain and Intent Classification

by   Pengfei Liu, et al.

User queries for a real-world dialog system may sometimes fall outside the scope of the system's capabilities, but appropriate system responses will enable smooth processing throughout the human-computer interaction. This paper is concerned with the user's intent, and focuses on out-of-scope intent classification in dialog systems. Although user intents are highly correlated with the application domain, few studies have exploited such correlations for intent classification. Rather than developing a two-stage approach that first classifies the domain and then the intent, we propose a hierarchical multi-task learning approach based on a joint model to classify domain and intent simultaneously. Novelties in the proposed approach include: (1) sharing supervised out-of-scope signals in joint modeling of domain and intent classification to replace a two-stage pipeline; and (2) introducing a hierarchical model that learns the intent and domain representations in the higher and lower layers respectively. Experiments show that the model outperforms existing methods in terms of accuracy, out-of-scope recall and F1. Additionally, threshold-based post-processing further improves performance by balancing precision and recall in intent classification.



There are no comments yet.


page 1

page 2

page 3

page 4


An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction

Task-oriented dialog systems need to know when a query falls outside the...

Are Pretrained Transformers Robust in Intent Classification? A Missing Ingredient in Evaluation of Out-of-Scope Intent Detection

Pretrained Transformer-based models were reported to be robust in intent...

Seq2Seq and Multi-Task Learning for joint intent and content extraction for domain specific interpreters

This study evaluates the performances of an LSTM network for detecting a...

HINT3: Raising the bar for Intent Detection in the Wild

Intent Detection systems in the real world are exposed to complexities o...

Bootstrapping NLU Models with Multi-task Learning

Bootstrapping natural language understanding (NLU) systems with minimal ...

The R-U-A-Robot Dataset: Helping Avoid Chatbot Deception by Detecting User Questions About Human or Non-Human Identity

Humans are increasingly interacting with machines through language, some...

A Base Camp for Scaling AI

Modern statistical machine learning (SML) methods share a major limitati...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.