Towards Multimodal Simultaneous Neural Machine Translation

04/07/2020
by   Aizhan Imankulova, et al.
0

Simultaneous translation involves translating a sentence before the speaker's utterance is completed in order to realize real-time understanding in multiple languages. This task is significantly harder than the general full sentence translation because of the shortage of input information during decoding. To alleviate this shortage, we propose multimodal simultaneous neural machine translation (MSNMT) which leverages visual information as an additional modality. Although the usefulness of images as an additional modality is moderate for full sentence translation, we verified, for the first time, its importance for simultaneous translation. Our experiments with the Multi30k dataset showed that MSNMT in a simultaneous setting significantly outperforms its text-only counterpart in situations where 5 or fewer input tokens are needed to begin translation. We then verified the importance of visual information during decoding by (a) performing an adversarial evaluation of MSNMT where we studied how models behave with incongruent input modality and (b) analyzing the image attention.

READ FULL TEXT

page 6

page 7

research
06/07/2016

Can neural machine translation do simultaneous translation?

We investigate the potential of attention-based neural machine translati...
research
09/21/2020

Generative Imagination Elevates Machine Translation

There are thousands of languages on earth, but visual perception is shar...
research
10/03/2016

Learning to Translate in Real-time with Neural Machine Translation

Translating in real-time, a.k.a. simultaneous translation, outputs trans...
research
03/19/2022

Neural Machine Translation with Phrase-Level Universal Visual Representations

Multimodal machine translation (MMT) aims to improve neural machine tran...
research
02/22/2021

Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translation

This paper addresses the problem of simultaneous machine translation (Si...
research
09/08/2021

Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models

Multimodal machine translation (MMT) systems have been shown to outperfo...
research
08/29/2019

Probing Representations Learned by Multimodal Recurrent and Transformer Models

Recent literature shows that large-scale language modeling provides exce...

Please sign up or login with your details

Forgot password? Click here to reset