Fast Nearest Neighbor Machine Translation

by   Yuxian Meng, et al.

Though nearest neighbor Machine Translation (kNN-MT) <cit.> has proved to introduce significant performance boosts over standard neural MT systems, it is prohibitively slow since it uses the entire reference corpus as the datastore for the nearest neighbor search. This means each step for each beam in the beam search has to search over the entire reference corpus. kNN-MT is thus two-order slower than vanilla MT models, making it hard to be applied to real-world applications, especially online services. In this work, we propose Fast kNN-MT to address this issue. Fast kNN-MT constructs a significantly smaller datastore for the nearest neighbor search: for each word in a source sentence, Fast kNN-MT first selects its nearest token-level neighbors, which is limited to tokens that are the same as the query token. Then at each decoding step, in contrast to using the entire corpus as the datastore, the search space is limited to target tokens corresponding to the previously selected reference source tokens. This strategy avoids search through the whole datastore for nearest neighbors and drastically improves decoding efficiency. Without loss of performance, Fast kNN-MT is two-order faster than kNN-MT, and is only two times slower than the standard NMT model. Fast kNN-MT enables the practical use of kNN-MT systems in real-world MT applications.[Code is available at <>]


page 1

page 2

page 3

page 4


Faster Nearest Neighbor Machine Translation

kNN based neural machine translation (kNN-MT) has achieved state-of-the-...

Adaptive Nearest Neighbor Machine Translation

kNN-MT, recently proposed by Khandelwal et al. (2020a), successfully com...

Nearest Neighbor Knowledge Distillation for Neural Machine Translation

k-nearest-neighbor machine translation (NN-MT), proposed by Khandelwal e...

Chunk-based Nearest Neighbor Machine Translation

Semi-parametric models, which augment generation with retrieval, have le...

Nearest Neighbor Machine Translation

We introduce k-nearest-neighbor machine translation (kNN-MT), which pred...

Learning Decoupled Retrieval Representation for Nearest Neighbour Neural Machine Translation

K-Nearest Neighbor Neural Machine Translation (kNN-MT) successfully inco...

compare-mt: A Tool for Holistic Comparison of Language Generation Systems

In this paper, we describe compare-mt, a tool for holistic analysis and ...

Code Repositories