Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems

12/29/2015

∙

We propose a simplified model of attention which is applicable to feed-forward neural networks and demonstrate that the resulting model can solve the synthetic "addition" and "multiplication" long-term memory problems for sequence lengths which are both longer and more widely varying than the best published results for these tasks.

READ FULL TEXT

Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems

Recurrent Dropout without Memory Loss

Working Memory for Online Memory Binding Tasks: A Hybrid Model

Direct Method for Training Feed-forward Neural Networks using Batch Extended Kalman Filter for Multi-Step-Ahead Predictions

Yeah, Right, Uh-Huh: A Deep Learning Backchannel Predictor

Predicting Memory Compiler Performance Outputs using Feed-Forward Neural Networks

Dynamic Move Tables and Long Branches with Backtracking in Computer Chess

An LSTM-based Plagiarism Detection via Attention Mechanism and a Population-based Approach for Pre-Training Parameters with imbalanced Classes

Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems

Related Research

Recurrent Dropout without Memory Loss

Working Memory for Online Memory Binding Tasks: A Hybrid Model

Direct Method for Training Feed-forward Neural Networks using Batch Extended Kalman Filter for Multi-Step-Ahead Predictions

Yeah, Right, Uh-Huh: A Deep Learning Backchannel Predictor

Predicting Memory Compiler Performance Outputs using Feed-Forward Neural Networks

Dynamic Move Tables and Long Branches with Backtracking in Computer Chess

An LSTM-based Plagiarism Detection via Attention Mechanism and a Population-based Approach for Pre-Training Parameters with imbalanced Classes