Nearest Neighbor Knowledge Distillation for Neural Machine Translation

05/01/2022
by   Zhixian Yang, et al.
0

k-nearest-neighbor machine translation (NN-MT), proposed by Khandelwal et al. (2021), has achieved many state-of-the-art results in machine translation tasks. Although effective, NN-MT requires conducting NN searches through the large datastore for each decoding step during inference, prohibitively increasing the decoding cost and thus leading to the difficulty for the deployment in real-world applications. In this paper, we propose to move the time-consuming NN search forward to the preprocessing phase, and then introduce Nearest Neighbor Knowledge Distillation (NN-KD) that trains the base NMT model to directly learn the knowledge of NN. Distilling knowledge retrieved by NN can encourage the NMT model to take more reasonable target tokens into consideration, thus addressing the overcorrection problem. Extensive experimental results show that, the proposed method achieves consistent improvement over the state-of-the-art baselines including NN-MT, while maintaining the same training and decoding speed as the standard NMT model.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/30/2021

Fast Nearest Neighbor Machine Translation

Though nearest neighbor Machine Translation (kNN-MT) <cit.> has proved t...
research
02/23/2023

Simple and Scalable Nearest Neighbor Machine Translation

kNN-MT is a straightforward yet powerful approach for fast domain adapta...
research
05/27/2021

Adaptive Nearest Neighbor Machine Translation

kNN-MT, recently proposed by Khandelwal et al. (2020a), successfully com...
research
10/17/2022

Towards Robust k-Nearest-Neighbor Machine Translation

k-Nearest-Neighbor Machine Translation (kNN-MT) becomes an important res...
research
06/10/2023

INK: Injecting kNN Knowledge in Nearest Neighbor Machine Translation

Neural machine translation has achieved promising results on many transl...
research
09/02/2019

An Adjusted Nearest Neighbor Algorithm Maximizing the F-Measure from Imbalanced Data

In this paper, we address the challenging problem of learning from imbal...
research
10/05/2020

On the Relevance of Cross-project Learning with Nearest Neighbours for Commit Message Generation

Commit messages play an important role in software maintenance and evolu...

Please sign up or login with your details

Forgot password? Click here to reset