Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translation

02/22/2021
by   Julia Ive, et al.
0

This paper addresses the problem of simultaneous machine translation (SiMT) by exploring two main concepts: (a) adaptive policies to learn a good trade-off between high translation quality and low latency; and (b) visual information to support this process by providing additional (visual) contextual information which may be available before the textual input is produced. For that, we propose a multimodal approach to simultaneous machine translation using reinforcement learning, with strategies to integrate visual and textual information in both the agent and the environment. We provide an exploration on how different types of visual information and integration strategies affect the quality and latency of simultaneous translation models, and demonstrate that visual cues lead to higher quality while keeping the latency low.

READ FULL TEXT
research
09/15/2020

Simultaneous Machine Translation with Visual Context

Simultaneous machine translation (SiMT) aims to translate a continuous i...
research
01/23/2022

Supervised Visual Attention for Simultaneous Multimodal Machine Translation

Recently, there has been a surge in research in multimodal machine trans...
research
04/18/2021

Stream-level Latency Evaluation for Simultaneous Machine Translation

Simultaneous machine translation has recently gained traction thanks to ...
research
04/07/2020

Towards Multimodal Simultaneous Neural Machine Translation

Simultaneous translation involves translating a sentence before the spea...
research
05/07/2018

Multimodal Machine Translation with Reinforcement Learning

Multimodal machine translation is one of the applications that integrate...
research
06/12/2019

Monotonic Infinite Lookback Attention for Simultaneous Machine Translation

Simultaneous machine translation begins to translate each source sentenc...
research
09/18/2020

Presenting Simultaneous Translation in Limited Space

Some methods of automatic simultaneous translation of a long-form speech...

Please sign up or login with your details

Forgot password? Click here to reset