Neuron Interaction Based Representation Composition for Neural Machine Translation

11/22/2019
by   Jian Li, et al.
0

Recent NLP studies reveal that substantial linguistic information can be attributed to single neurons, i.e., individual dimensions of the representation vectors. We hypothesize that modeling strong interactions among neurons helps to better capture complex information by composing the linguistic properties embedded in individual neurons. Starting from this intuition, we propose a novel approach to compose representations learned by different components in neural machine translation (e.g., multi-layer networks or multi-head attention), based on modeling strong interactions among neurons in the representation vectors. Specifically, we leverage bilinear pooling to model pairwise multiplicative interactions among individual neurons, and a low-rank approximation to make the model computationally feasible. We further propose extended bilinear pooling to incorporate first-order representations. Experiments on WMT14 English-German and English-French translation tasks show that our model consistently improves performances over the SOTA Transformer baseline. Further analyses demonstrate that our approach indeed captures more syntactic and semantic information as expected.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/03/2018

Identifying and Controlling Important Neurons in Neural Machine Translation

Neural machine translation (NMT) models learn representations containing...
research
02/16/2020

Multi-layer Representation Fusion for Neural Machine Translation

Neural machine translation systems require a number of stacked layers fo...
research
10/06/2021

On Neurons Invariant to Sentence Structural Changes in Neural Machine Translation

To gain insight into the role neurons play, we study the activation patt...
research
06/04/2019

Exploiting Sentential Context for Neural Machine Translation

In this work, we present novel approaches to exploit sentential context ...
research
12/21/2018

What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models

Despite the remarkable evolution of deep neural networks in natural lang...
research
06/25/2020

Learning Source Phrase Representations for Neural Machine Translation

The Transformer translation model (Vaswani et al., 2017) based on a mult...
research
12/05/2019

Pairwise Neural Machine Translation Evaluation

We present a novel framework for machine translation evaluation using ne...

Please sign up or login with your details

Forgot password? Click here to reset