Warped Language Models for Noise Robust Language Understanding

by   Mahdi Namazifar, et al.

Masked Language Models (MLM) are self-supervised neural networks trained to fill in the blanks in a given sentence with masked tokens. Despite the tremendous success of MLMs for various text based tasks, they are not robust for spoken language understanding, especially for spontaneous conversational speech recognition noise. In this work we introduce Warped Language Models (WLM) in which input sentences at training time go through the same modifications as in MLM, plus two additional modifications, namely inserting and dropping random tokens. These two modifications extend and contract the sentence in addition to the modifications in MLMs, hence the word "warped" in the name. The insertion and drop modification of the input text during training of WLM resemble the types of noise due to Automatic Speech Recognition (ASR) errors, and as a result WLMs are likely to be more robust to ASR noise. Through computational results we show that natural language understanding systems built on top of WLMs perform better compared to those built based on MLMs, especially in the presence of ASR errors.


page 1

page 2

page 3

page 4


Building Robust Spoken Language Understanding by Cross Attention between Phoneme Sequence and ASR Hypothesis

Building Spoken Language Understanding (SLU) robust to Automatic Speech ...

Do We Still Need Automatic Speech Recognition for Spoken Language Understanding?

Spoken language understanding (SLU) tasks are usually solved by first tr...

ASR-GLUE: A New Multi-task Benchmark for ASR-Robust Natural Language Understanding

Language understanding in speech-based systems have attracted much atten...

Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining

Much recent work on Spoken Language Understanding (SLU) is limited in at...

Personalized Query Rewriting in Conversational AI Agents

Spoken language understanding (SLU) systems in conversational AI agents ...

Towards Semi-Supervised Semantics Understanding from Speech

Much recent work on Spoken Language Understanding (SLU) falls short in a...

RecoBERT: A Catalog Language Model for Text-Based Recommendations

Language models that utilize extensive self-supervised pre-training from...