Kaizen: Continuously improving teacher using Exponential Moving Average for semi-supervised speech recognition

by   Vimal Manohar, et al.

In this paper, we introduce the Kaizen framework that uses a continuously improving teacher to generate pseudo-labels for semi-supervised training. The proposed approach uses a teacher model which is updated as the exponential moving average of the student model parameters. This can be seen as a continuous version of the iterative pseudo-labeling approach for semi-supervised training. It is applicable for different training criteria, and in this paper we demonstrate it for frame-level hybrid hidden Markov model - deep neural network (HMM-DNN) models and sequence-level connectionist temporal classification (CTC) based models. The proposed approach shows more than 10 word error rate (WER) reduction over standard teacher-student training and more than 50% relative WER reduction over 10 hour supervised baseline when using large scale realistic unsupervised public videos in UK English and Italian languages.



There are no comments yet.


page 1

page 2

page 3

page 4


Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning

We present a plug-in replacement for batch normalization (BN) called exp...

Spatial Ensemble: a Novel Model Smoothing Mechanism for Student-Teacher Framework

Model smoothing is of central importance for obtaining a reliable teache...

Snowball: Iterative Model Evolution and Confident Sample Discovery for Semi-Supervised Learning on Very Small Labeled Datasets

In this work, we develop a joint sample discovery and iterative model ev...

Teacher-Student MixIT for Unsupervised and Semi-supervised Speech Separation

In this paper, we introduce a novel semi-supervised learning framework f...

Self-Training with Differentiable Teacher

Self-training achieves enormous success in various semi-supervised and w...

Teacher-Student Asynchronous Learning with Multi-Source Consistency for Facial Landmark Detection

Due to the high annotation cost of large-scale facial landmark detection...

Teacher-Student chain for efficient semi-supervised histology image classification

Deep learning shows great potential for the domain of digital pathology....
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.