On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation

05/07/2020
by   Chaojun Wang, et al.
Universität Zürich
0

The standard training algorithm in neural machine translation (NMT) suffers from exposure bias, and alternative algorithms have been proposed to mitigate this. However, the practical impact of exposure bias is under debate. In this paper, we link exposure bias to another well-known problem in NMT, namely the tendency to generate hallucinations under domain shift. In experiments on three datasets with multiple test domains, we show that exposure bias is partially to blame for hallucinations, and that training with Minimum Risk Training, which avoids exposure bias, can mitigate this. Our analysis explains why exposure bias is more problematic under domain shift, and also links exposure bias to the beam search problem, i.e. performance deterioration with increasing beam size. Our results provide a new justification for methods that reduce exposure bias: even if they do not increase performance on in-domain test sets, they can increase model robustness to domain shift.

READ FULL TEXT

page 1

page 2

page 3

page 4

05/18/2021

Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translation

Neural Machine Translation (NMT) currently exhibits biases such as produ...
09/10/2018

Greedy Search with Probabilistic N-gram Matching for Neural Machine Translation

Neural machine translation (NMT) models are usually trained with the wor...
08/29/2018

Correcting Length Bias in Neural Machine Translation

We study two problems in neural machine translation (NMT). First, in bea...
09/13/2021

Multi-Sentence Resampling: A Simple Approach to Alleviate Dataset Length Bias and Beam-Search Degradation

Neural Machine Translation (NMT) is known to suffer from a beam-search p...
08/27/2019

On NMT Search Errors and Model Errors: Cat Got Your Tongue?

We report on search errors and model errors in neural machine translatio...
09/09/2021

Fixing exposure bias with imitation learning needs powerful oracles

We apply imitation learning (IL) to tackle the NMT exposure bias problem...
09/17/2021

Relating Neural Text Degeneration to Exposure Bias

This work focuses on relating two mysteries in neural-based text generat...

Code Repositories

Exposure-Bias-Hallucination-Domain-Shift

None


view repo

Please sign up or login with your details

Forgot password? Click here to reset