Multilingual and crosslingual speech recognition using phonological-vector based phone embeddings

by   Chengrui Zhu, et al.

The use of phonological features (PFs) potentially allows language-specific phones to remain linked in training, which is highly desirable for information sharing for multilingual and crosslingual speech recognition methods for low-resourced languages. A drawback suffered by previous methods in using phonological features is that the acoustic-to-PF extraction in a bottom-up way is itself difficult. In this paper, we propose to join phonology driven phone embedding (top-down) and deep neural network (DNN) based acoustic feature extraction (bottom-up) to calculate phone probabilities. The new method is called JoinAP (Joining of Acoustics and Phonology). Remarkably, no inversion from acoustics to phonological features is required for speech recognition. For each phone in the IPA (International Phonetic Alphabet) table, we encode its phonological features to a phonological-vector, and then apply linear or nonlinear transformation of the phonological-vector to obtain the phone embedding. A series of multilingual and crosslingual (both zero-shot and few-shot) speech recognition experiments are conducted on the CommonVoice dataset (German, French, Spanish and Italian) and the AISHLL-1 dataset (Mandarin), and demonstrate the superiority of JoinAP with nonlinear phone embeddings over both JoinAP with linear phone embeddings and the traditional method with flat phone embeddings.


page 1

page 2

page 3

page 4


A multilingual approach to joint Speech and Accent Recognition with DNN-HMM framework

Human can perform multi-task recognition from speech. For instance, huma...

Boosting Local Spectro-Temporal Features for Speech Analysis

We introduce the problem of phone classification in the context of speec...

Common Phone: A Multilingual Dataset for Robust Acoustic Modelling

Current state of the art acoustic models can easily comprise more than 1...

Calibration of Phone Likelihoods in Automatic Speech Recognition

In this paper we study the probabilistic properties of the posteriors in...

Phoneme Recognition through Fine Tuning of Phonetic Representations: a Case Study on Luhya Language Varieties

Models pre-trained on multiple languages have shown significant promise ...

Multilingual and Multimode Phone Recognition System for Indian Languages

The aim of this paper is to develop a flexible framework capable of auto...

Towards Automated Assessment of Stuttering and Stuttering Therapy

Stuttering is a complex speech disorder that can be identified by repeti...

Please sign up or login with your details

Forgot password? Click here to reset