Alleviating the Inequality of Attention Heads for Neural Machine Translation

09/21/2020

∙

by   Zewei Sun, et al.

∙

0

∙

share

Recent studies show that the attention heads in Transformer are not equal. We relate this phenomenon to the imbalance training of multi-head attention and the model dependence on specific heads. To tackle this problem, we propose a simple masking method: HeadMask, in two specific ways. Experiments show that translation improvements are achieved on multiple language pairs. Subsequent empirical analyses also support our assumption and confirm the effectiveness of the method.