Enriched Pre-trained Transformers for Joint Slot Filling and Intent Detection

04/30/2020
by   Momchil Hardalov, et al.
0

Detecting the user's intent and finding the corresponding slots among the utterance's words are important tasks in natural language understanding. Their interconnected nature makes their joint modeling a standard part of training such models. Moreover, data scarceness and specialized vocabularies pose additional challenges. Recently, the advances in pre-trained language models, namely contextualized models such as ELMo and BERT have revolutionized the field by tapping the potential of training very large models with just a few steps of fine-tuning on a task-specific dataset. Here, we leverage such model, namely BERT, and we design a novel architecture on top it. Moreover, we propose an intent pooling attention mechanism, and we reinforce the slot filling task by fusing intent distributions, word features, and token representations. The experimental results on standard datasets show that our model outperforms both the current non-BERT state of the art as well as some stronger BERT-based baselines.

READ FULL TEXT
research
02/28/2019

BERT for Joint Intent Classification and Slot Filling

Intent classification and slot filling are two essential tasks for natur...
research
04/21/2020

DIET: Lightweight Language Understanding for Dialogue Systems

Large-scale pre-trained language models have shown impressive results on...
research
05/28/2023

Tri-level Joint Natural Language Understanding for Multi-turn Conversational Datasets

Natural language understanding typically maps single utterances to a dua...
research
07/05/2019

Multi-lingual Intent Detection and Slot Filling in a Joint BERT-based Model

Intent Detection and Slot Filling are two pillar tasks in Spoken Natural...
research
10/07/2022

A Unified Framework for Multi-intent Spoken Language Understanding with prompting

Multi-intent Spoken Language Understanding has great potential for wides...
research
09/08/2020

Simple is Better! Lightweight Data Augmentation for Low Resource Slot Filling and Intent Classification

Neural-based models have achieved outstanding performance on slot fillin...

Please sign up or login with your details

Forgot password? Click here to reset