On a novel training algorithm for sequence-to-sequence predictive recurrent networks

06/27/2021
by   Boris Rubinstein, et al.
0

Neural networks mapping sequences to sequences (seq2seq) lead to significant progress in machine translation and speech recognition. Their traditional architecture includes two recurrent networks (RNs) followed by a linear predictor. In this manuscript we perform analysis of a corresponding algorithm and show that the parameters of the RNs of the well trained predictive network are not independent of each other. Their dependence can be used to significantly improve the network effectiveness. The traditional seq2seq algorithms require short term memory of a size proportional to the predicted sequence length. This requirement is quite difficult to implement in a neuroscience context. We present a novel memoryless algorithm for seq2seq predictive networks and compare it to the traditional one in the context of time series prediction. We show that the new algorithm is more robust and makes predictions with higher accuracy than the traditional one.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/05/2020

A fast memoryless predictive algorithm in a chain of recurrent neural networks

In the recent publication (arxiv:2007.08063v2 [cs.LG]) a fast prediction...
research
07/02/2018

Dynamic Prediction Length for Time Series with Sequence to Sequence Networks

Recurrent neural networks and sequence to sequence models require a pred...
research
06/09/2022

It's a super deal – train recurrent network on noisy data and get smooth prediction free

Recent research demonstrate that prediction of time series by predictive...
research
11/23/2018

A Hierarchical Neural Network for Sequence-to-Sequences Learning

In recent years, the sequence-to-sequence learning neural networks with ...
research
05/29/2022

The impact of memory on learning sequence-to-sequence tasks

The recent success of neural networks in machine translation and other f...
research
06/23/2020

Thalamocortical motor circuit insights for more robust hierarchical control of complex sequences

We study learning of recurrent neural networks that produce temporal seq...
research
07/21/2023

On the Universality of Linear Recurrences Followed by Nonlinear Projections

In this note (work in progress towards a full-length paper) we show that...

Please sign up or login with your details

Forgot password? Click here to reset