DeepAI AI Chat
Log In Sign Up

Recurrent Neural Network Training with Dark Knowledge Transfer

05/18/2015
by   Zhiyuan Tang, et al.
Tsinghua University
0

Recurrent neural networks (RNNs), particularly long short-term memory (LSTM), have gained much attention in automatic speech recognition (ASR). Although some successful stories have been reported, training RNNs remains highly challenging, especially with limited training data. Recent research found that a well-trained model can be used as a teacher to train other child models, by using the predictions generated by the teacher model as supervision. This knowledge transfer learning has been employed to train simple neural nets with a complex one, so that the final performance can reach a level that is infeasible to obtain by regular training. In this paper, we employ the knowledge transfer learning approach to train RNNs (precisely LSTM) using a deep neural network (DNN) model as the teacher. This is different from most of the existing research on knowledge transfer learning, since the teacher (DNN) is assumed to be weaker than the child (RNN); however, our experiments on an ASR task showed that it works fairly well: without applying any tricks on the learning scheme, this approach can train RNNs successfully even with limited training data.

READ FULL TEXT

page 1

page 2

page 3

page 4

02/20/2021

Exploring Knowledge Distillation of a Deep Neural Network for Multi-Script identification

Multi-lingual script identification is a difficult task consisting of di...
06/07/2015

Knowledge Transfer Pre-training

Pre-training is crucial for learning deep neural networks. Most of exist...
04/07/2015

Transferring Knowledge from a RNN to a DNN

Deep Neural Network (DNN) acoustic models have yielded many state-of-the...
11/14/2018

Performance Estimation of Synthesis Flows cross Technologies using LSTMs and Transfer Learning

Due to the increasing complexity of Integrated Circuits (ICs) and System...
05/01/2020

Can a powerful neural network be a teacher for a weaker neural network?

The transfer learning technique is widely used to learning in one contex...
05/21/2020

An approach to Beethoven's 10th Symphony

Ludwig van Beethoven composed his symphonies between 1799 and 1825, when...
07/23/2015

Neural NILM: Deep Neural Networks Applied to Energy Disaggregation

Energy disaggregation estimates appliance-by-appliance electricity consu...