Log In Sign Up

Learning Audio-Text Agreement for Open-vocabulary Keyword Spotting

by   Hyeon-Kyeong Shin, et al.

In this paper, we propose a novel end-to-end user-defined keyword spotting method that utilizes linguistically corresponding patterns between speech and text sequences. Unlike previous approaches requiring speech keyword enrollment, our method compares input queries with an enrolled text keyword sequence. To place the audio and text representations within a common latent space, we adopt an attention-based cross-modal matching approach that is trained in an end-to-end manner with monotonic matching loss and keyword classification loss. We also utilize a de-noising loss for the acoustic embedding network to improve robustness in noisy environments. Additionally, we introduce the LibriPhrase dataset, a new short-phrase dataset based on LibriSpeech for efficiently training keyword spotting models. Our proposed method achieves competitive results on various evaluation sets compared to other single-modal and cross-modal baselines.


page 2

page 4


Tie Your Embeddings Down: Cross-Modal Latent Spaces for End-to-end Spoken Language Understanding

End-to-end (E2E) spoken language understanding (SLU) systems can infer t...

Deep Latent Space Learning for Cross-modal Mapping of Audio and Visual Signals

We propose a novel deep training algorithm for joint representation of a...

Speech Augmentation Based Unsupervised Learning for Keyword Spotting

In this paper, we investigated a speech augmentation based unsupervised ...

Cross-modal Contrastive Learning for Speech Translation

How can we learn unified representations for spoken utterances and their...

Visual Keyword Spotting with Attention

In this paper, we consider the task of spotting spoken keywords in silen...

Embedding Compression for Text Classification Using Dictionary Screening

In this paper, we propose a dictionary screening method for embedding co...

Image2Reverb: Cross-Modal Reverb Impulse Response Synthesis

Measuring the acoustic characteristics of a space is often done by captu...