A Primal-Dual Method for Training Recurrent Neural Networks Constrained by the Echo-State Property

11/24/2013
by   Jianshu Chen, et al.
0

We present an architecture of a recurrent neural network (RNN) with a fully-connected deep neural network (DNN) as its feature extractor. The RNN is equipped with both causal temporal prediction and non-causal look-ahead, via auto-regression (AR) and moving-average (MA), respectively. The focus of this paper is a primal-dual training method that formulates the learning of the RNN as a formal optimization problem with an inequality constraint that provides a sufficient condition for the stability of the network dynamics. Experimental results demonstrate the effectiveness of this new method, which achieves 18.86 phone recognition error on the TIMIT benchmark for the core test set. The result approaches the best result of 17.7 with long short-term memory (LSTM). The results also show that the proposed primal-dual training method produces lower recognition errors than the popular RNN methods developed earlier based on the carefully tuned threshold parameter that heuristically prevents the gradient from exploding.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/30/2019

Two-phase flow regime prediction using LSTM based deep recurrent neural network

Long short-term memory (LSTM) and recurrent neural network (RNN) has ach...
research
02/14/2020

Real-time speech enhancement using equilibriated RNN

We propose a speech enhancement method using a causal deep neural networ...
research
04/28/2020

How Chaotic Are Recurrent Neural Networks?

Recurrent neural networks (RNNs) are non-linear dynamic systems. Previou...
research
11/05/2020

Alignment Restricted Streaming Recurrent Neural Network Transducer

There is a growing interest in the speech community in developing Recurr...
research
07/11/2017

Adversarial Generation of Training Examples: Applications to Moving Vehicle License Plate Recognition

Generative Adversarial Networks (GAN) have attracted much research atten...
research
06/27/2023

Recurrent Neural Network-coupled SPAD TCSPC System for Real-time Fluorescence Lifetime Imaging

Fluorescence lifetime imaging (FLI) has been receiving increased attenti...

Please sign up or login with your details

Forgot password? Click here to reset