On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation

by   Chaojun Wang, et al.
Universität Zürich

The standard training algorithm in neural machine translation (NMT) suffers from exposure bias, and alternative algorithms have been proposed to mitigate this. However, the practical impact of exposure bias is under debate. In this paper, we link exposure bias to another well-known problem in NMT, namely the tendency to generate hallucinations under domain shift. In experiments on three datasets with multiple test domains, we show that exposure bias is partially to blame for hallucinations, and that training with Minimum Risk Training, which avoids exposure bias, can mitigate this. Our analysis explains why exposure bias is more problematic under domain shift, and also links exposure bias to the beam search problem, i.e. performance deterioration with increasing beam size. Our results provide a new justification for methods that reduce exposure bias: even if they do not increase performance on in-domain test sets, they can increase model robustness to domain shift.


page 1

page 2

page 3

page 4


Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translation

Neural Machine Translation (NMT) currently exhibits biases such as produ...

Greedy Search with Probabilistic N-gram Matching for Neural Machine Translation

Neural machine translation (NMT) models are usually trained with the wor...

Correcting Length Bias in Neural Machine Translation

We study two problems in neural machine translation (NMT). First, in bea...

Multi-Sentence Resampling: A Simple Approach to Alleviate Dataset Length Bias and Beam-Search Degradation

Neural Machine Translation (NMT) is known to suffer from a beam-search p...

On NMT Search Errors and Model Errors: Cat Got Your Tongue?

We report on search errors and model errors in neural machine translatio...

Fixing exposure bias with imitation learning needs powerful oracles

We apply imitation learning (IL) to tackle the NMT exposure bias problem...

Relating Neural Text Degeneration to Exposure Bias

This work focuses on relating two mysteries in neural-based text generat...

Code Repositories



view repo

Please sign up or login with your details

Forgot password? Click here to reset