Numpy-RNNs
None
view repo
Learning long term dependencies in recurrent networks is difficult due to vanishing and exploding gradients. To overcome this difficulty, researchers have developed sophisticated optimization techniques and network architectures. In this paper, we propose a simpler solution that use recurrent neural networks composed of rectified linear units. Key to our solution is the use of the identity matrix or its scaled version to initialize the recurrent weight matrix. We find that our solution is comparable to LSTM on our four benchmarks: two toy problems involving long-range temporal structures, a large language modeling problem and a benchmark speech recognition problem.
READ FULL TEXT
In recent years significant progress has been made in successfully train...
read it
Recurrent neural network is a powerful model that learns temporal patter...
read it
Recurrent networks have achieved great success on various sequential tas...
read it
It is a known fact that training recurrent neural networks for tasks tha...
read it
It is well known that it is challenging to train deep neural networks an...
read it
Common recurrent neural network architectures scale poorly due to the
in...
read it
Recurrent neural networks (RNNs) are notoriously difficult to train. Whe...
read it
None
Comments
There are no comments yet.