Are Pretrained Transformers Robust in Intent Classification? A Missing Ingredient in Evaluation of Out-of-Scope Intent Detection

06/08/2021
by   Jian-Guo Zhang, et al.
19

Pretrained Transformer-based models were reported to be robust in intent classification. In this work, we first point out the importance of in-domain out-of-scope detection in few-shot intent recognition tasks and then illustrate the vulnerability of pretrained Transformer-based models against samples that are in-domain but out-of-scope (ID-OOS). We empirically show that pretrained models do not perform well on both ID-OOS examples and general out-of-scope examples, especially on fine-grained few-shot intent detection tasks. To figure out how the models mistakenly classify ID-OOS intents as in-scope intents, we further conduct analysis on confidence scores and the overlapping keywords and provide several prospective directions for future work. We release the relevant resources to facilitate future research.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/21/2021

ConvFiT: Conversational Fine-Tuning of Pretrained Language Models

Transformer-based language models (LMs) pretrained on large text collect...
research
05/18/2022

Exploring the Advantages of Dense-Vector to One-Hot Encoding of Intent Classes in Out-of-Scope Detection Tasks

This work explores the intrinsic limitations of the popular one-hot enco...
research
04/30/2021

Hierarchical Modeling for Out-of-Scope Domain and Intent Classification

User queries for a real-world dialog system may sometimes fall outside t...
research
03/10/2020

Efficient Intent Detection with Dual Sentence Encoders

Building conversational systems in new domains and with added functional...
research
05/06/2022

Fine-grained Intent Classification in the Legal Domain

A law practitioner has to go through a lot of long legal case proceeding...
research
09/04/2019

An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction

Task-oriented dialog systems need to know when a query falls outside the...
research
05/18/2021

Wizard of Search Engine: Access to Information Through Conversations with Search Engines

Conversational information seeking (CIS) is playing an increasingly impo...

Please sign up or login with your details

Forgot password? Click here to reset