-
ST-BERT: Cross-modal Language Model Pre-training For End-to-end Spoken Language Understanding
Language model pre-training has shown promising results in various downs...
read it
-
Train No Evil: Selective Masking for Task-guided Pre-training
Recently, pre-trained language models mostly follow the pre-training-the...
read it
-
LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning
While pre-training and fine-tuning, e.g., BERT <cit.>, GPT-2 <cit.>, hav...
read it
-
What does BERT know about books, movies and music? Probing BERT for Conversational Recommendation
Heavily pre-trained transformer models such as BERT have recently shown ...
read it
-
Pre-Training for Query Rewriting in A Spoken Language Understanding System
Query rewriting (QR) is an increasingly important technique to reduce cu...
read it
-
Unsupervised Transfer Learning for Spoken Language Understanding in Intelligent Agents
User interaction with voice-powered agents generates large amounts of un...
read it
-
Hierarchical Pre-training for Sequence Labelling in Spoken Dialog
Sequence labelling tasks like Dialog Act and Emotion/Sentiment identific...
read it
Style Attuned Pre-training and Parameter Efficient Fine-tuning for Spoken Language Understanding
Neural models have yielded state-of-the-art results in deciphering spoken language understanding (SLU) problems; however, these models require a significant amount of domain-specific labeled examples for training, which is prohibitively expensive. While pre-trained language models like BERT have been shown to capture a massive amount of knowledge by learning from unlabeled corpora and solve SLU using fewer labeled examples for adaption, the encoding of knowledge is implicit and agnostic to downstream tasks. Such encoding results in model inefficiencies in parameter usage: an entirely new model is required for every domain. To address these challenges, we introduce a novel SLU framework, comprising a conversational language modeling (CLM) pre-training task and a light encoder architecture. The CLM pre-training enables networks to capture the representation of the language in conversation style with the presence of ASR errors. The light encoder architecture separates the shared pre-trained networks from the mappings of generally encoded knowledge to specific domains of SLU, allowing for the domain adaptation to be performed solely at the light encoder and thus increasing efficiency. With the framework, we match the performance of state-of-the-art SLU results on Alexa internal datasets and on two public ones (ATIS, SNIPS), adding only 4.4 task.
READ FULL TEXT
Comments
There are no comments yet.