Ensembling of Distilled Models from Multi-task Teachers for Constrained Resource Language Pairs

11/26/2021

∙

by   Amr Hendy, et al.

∙

0

∙

share

This paper describes our submission to the constrained track of WMT21 shared news translation task. We focus on the three relatively low resource language pairs Bengali to and from Hindi, English to and from Hausa, and Xhosa to and from Zulu. To overcome the limitation of relatively low parallel data we train a multilingual model using a multitask objective employing both parallel and monolingual data. In addition, we augment the data using back translation. We also train a bilingual model incorporating back translation and knowledge distillation then combine the two models using sequence-to-sequence mapping. We see around 70 around 25 to and from Zulu compared to bilingual baselines.