Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction

08/11/2018
by   Maha Elbayad, et al.
0

Current state-of-the-art machine translation systems are based on encoder-decoder architectures, that first encode the input sequence, and then generate an output sequence based on the input encoding. Both are interfaced with an attention mechanism that recombines a fixed encoding of the source tokens based on the decoder state. We propose an alternative approach which instead relies on a single 2D convolutional neural network across both sequences. Each layer of our network re-codes source tokens on the basis of the output sequence produced so far. Attention-like properties are therefore pervasive throughout the network. Our model yields excellent results, outperforming state-of-the-art encoder-decoder systems, while being conceptually simpler and having fewer parameters.

READ FULL TEXT
research
01/11/2021

Context- and Sequence-Aware Convolutional Recurrent Encoder for Neural Machine Translation

Neural Machine Translation model is a sequence-to-sequence converter bas...
research
07/01/2017

Efficient Attention using a Fixed-Size Memory Representation

The standard content-based attention mechanism typically used in sequenc...
research
08/29/2019

Translating Mathematical Formula Images to LaTeX Sequences Using Deep Neural Networks with Sequence-level Training

In this paper we propose a deep neural network model with an encoder-dec...
research
05/09/2018

A Click Sequence Model for Web Search

Getting a better understanding of user behavior is important for advanci...
research
06/13/2018

Double Path Networks for Sequence to Sequence Learning

Encoder-decoder based Sequence to Sequence learning (S2S) has made remar...
research
09/17/2018

Quantum Statistics-Inspired Neural Attention

Sequence-to-sequence (encoder-decoder) models with attention constitute ...
research
12/01/2017

Folded Recurrent Neural Networks for Future Video Prediction

Future video prediction is an ill-posed Computer Vision problem that rec...

Please sign up or login with your details

Forgot password? Click here to reset