Towards Linear Time Neural Machine Translation with Capsule Networks

11/01/2018
by   Mingxuan Wang, et al.
0

In this study, we first investigate a novel capsule network with dynamic routing for linear time Neural Machine Translation (NMT), referred as CapsNMT. CapsNMT uses an aggregation mechanism to map the source sentence into a matrix with pre-determined size, and then applys a deep LSTM network to decode the target sequence from the source representation. Unlike the previous work sutskever2014sequence to store the source sentence with a passive and bottom-up way, the dynamic routing policy encodes the source sentence with an iterative process to decide the credit attribution between nodes from lower and higher layers. CapsNMT has two core properties: it runs in time that is linear in the length of the sequences and provides a more flexible way to select, represent and aggregates the part-whole information of the source sentence. On WMT14 English-German task and a larger WMT14 English-French task, CapsNMT achieves comparable results with the state-of-the-art NMT systems. To the best of our knowledge, this is the first work that capsule networks have been empirically investigated for sequence to sequence problems.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/31/2016

Neural Machine Translation in Linear Time

We present a novel neural network for processing sequences. The ByteNet ...
research
12/27/2019

Explicit Sentence Compression for Neural Machine Translation

State-of-the-art Transformer-based neural machine translation (NMT) syst...
research
08/29/2018

An Operation Sequence Model for Explainable Neural Machine Translation

We propose to achieve explainable neural machine translation (NMT) by ch...
research
02/15/2019

Dynamic Layer Aggregation for Neural Machine Translation with Routing-by-Agreement

With the promising progress of deep neural networks, layer aggregation h...
research
06/05/2018

Information Aggregation via Dynamic Routing for Sequence Encoding

While much progress has been made in how to encode a text sequence into ...
research
07/23/2020

Sequential Routing Framework: Fully Capsule Network-based Speech Recognition

Capsule networks (CapsNets) have recently gotten attention as alternativ...
research
08/31/2019

Improving Multi-Head Attention with Capsule Networks

Multi-head attention advances neural machine translation by working out ...

Please sign up or login with your details

Forgot password? Click here to reset