An Intrinsic Nearest Neighbor Analysis of Neural Machine Translation Architectures

07/08/2019
by   Hamidreza Ghader, et al.
1

Earlier approaches indirectly studied the information captured by the hidden states of recurrent and non-recurrent neural machine translation models by feeding them into different classifiers. In this paper, we look at the encoder hidden states of both transformer and recurrent machine translation models from the nearest neighbors perspective. We investigate to what extent the nearest neighbors share information with the underlying word embeddings as well as related WordNet entries. Additionally, we study the underlying syntactic structure of the nearest neighbors to shed light on the role of syntactic similarities in bringing the neighbors together. We compare transformer and recurrent models in a more intrinsic way in terms of capturing lexical semantics and syntactic structures, in contrast to extrinsic approaches used by previous works. In agreement with the extrinsic evaluations in the earlier works, our experimental results show that transformers are superior in capturing lexical semantics, but not necessarily better in capturing the underlying syntax. Additionally, we show that the backward recurrent layer in a recurrent model learns more about the semantics of words, whereas the forward recurrent layer encodes more context.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/13/2021

Semantics-aware Attention Improves Neural Machine Translation

The integration of syntactic structures into Transformer machine transla...
research
01/11/2021

Context- and Sequence-Aware Convolutional Recurrent Encoder for Neural Machine Translation

Neural Machine Translation model is a sequence-to-sequence converter bas...
research
08/19/2019

Recurrent Graph Syntax Encoder for Neural Machine Translation

Syntax-incorporated machine translation models have been proven successf...
research
10/05/2019

How Transformer Revitalizes Character-based Neural Machine Translation: An Investigation on Japanese-Vietnamese Translation Systems

While translating between Chinese-centric languages, many works have dis...
research
10/03/2017

Improving Lexical Choice in Neural Machine Translation

We explore two solutions to the problem of mistranslating rare words in ...
research
08/30/2019

Encoders Help You Disambiguate Word Senses in Neural Machine Translation

Neural machine translation (NMT) has achieved new state-of-the-art perfo...
research
07/09/2017

Neural Machine Translation between Herbal Prescriptions and Diseases

The current study applies deep learning to herbalism. Toward the goal, w...

Please sign up or login with your details

Forgot password? Click here to reset