Prosody-Aware Neural Machine Translation for Dubbing

12/16/2021
by   Derek Tam, et al.
0

We introduce the task of prosody-aware machine translation which aims at generating translations suitable for dubbing. Dubbing of a spoken sentence requires transferring the content as well as the prosodic structure of the source into the target language to preserve timing information. Practically, this implies correctly projecting pauses from the source to the target and ensuring that target speech segments have roughly the same duration of the corresponding source segments. In this work, we propose an implicit and explicit modeling approaches to integrate prosody information into neural machine translation. Experiments on English-German/French with automatic metrics show that the simplest of the considered approaches works best. Results are confirmed by human evaluations of translations and dubbed videos.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/25/2016

Variational Neural Machine Translation

Models of neural machine translation are often from a discriminative fam...
research
11/10/2017

Document Context Neural Machine Translation with Memory Networks

We present a document-level neural machine translation model which takes...
research
11/20/2019

Controlling Neural Machine Translation Formality with Synthetic Supervision

This work aims to produce translations that convey source language conte...
research
11/24/2020

Two-Way Neural Machine Translation: A Proof of Concept for Bidirectional Translation Modeling using a Two-Dimensional Grid

Neural translation models have proven to be effective in capturing suffi...
research
10/08/2021

Machine Translation Verbosity Control for Automatic Dubbing

Automatic dubbing aims at seamlessly replacing the speech in a video doc...
research
05/22/2023

Improving Isochronous Machine Translation with Target Factors and Auxiliary Counters

To translate speech for automatic dubbing, machine translation needs to ...
research
03/03/2022

As Little as Possible, as Much as Necessary: Detecting Over- and Undertranslations with Contrastive Conditioning

Omission and addition of content is a typical issue in neural machine tr...

Please sign up or login with your details

Forgot password? Click here to reset