Self-Attention with Cross-Lingual Position Representation

by   Liang Ding, et al.

Position encoding (PE), an essential part of self-attention networks (SANs), is used to preserve the word order information for natural language processing tasks, generating fixed position indices for input sequences. However, in cross-lingual scenarios, machine translation, the PEs of source and target sentences are modeled independently. Due to word order divergences in different languages, modeling the cross-lingual positional relationships might help SANs tackle this problem. In this paper, we augment SANs with cross-lingual position representations to model the bilingually aware latent structure for the input sentence. Specifically, we utilize bracketing transduction grammar (BTG)-based reordering information to encourage SANs to learn bilingual diagonal alignments. Experimental results on WMT'14 EnglishGerman, WAT'17 JapaneseEnglish, and WMT'17 ChineseEnglish translation tasks demonstrate that our approach significantly and consistently improves translation quality over strong baselines. Extensive analyses confirm that the performance gains come from the cross-lingual information.


page 1

page 2

page 3

page 4


Do We Need Word Order Information for Cross-lingual Sequence Labeling

Most of the recent work in cross-lingual adaptation does not consider th...

Enhancing Cross-lingual Transfer by Manifold Mixup

Based on large-scale pre-trained multilingual representations, recent cr...

On Learning Universal Representations Across Languages

Recent studies have demonstrated the overwhelming advantage of cross-lin...

Towards Debiasing Translation Artifacts

Cross-lingual natural language processing relies on translation, either ...

Assessing the Ability of Self-Attention Networks to Learn Word Order

Self-attention networks (SAN) have attracted a lot of interests due to t...

Unpaired Cross-lingual Image Caption Generation with Self-Supervised Rewards

Generating image descriptions in different languages is essential to sat...

Improving cross-lingual model transfer by chunking

We present a shallow parser guided cross-lingual model transfer approach...