From Speech-to-Speech Translation to Automatic Dubbing

by   Marcello Federico, et al.

We present enhancements to a speech-to-speech translation pipeline in order to perform automatic dubbing. Our architecture features neural machine translation generating output of preferred length, prosodic alignment of the translation with the original speech segments, neural text-to-speech with fine tuning of the duration of each utterance, and, finally, audio rendering to enriches text-to-speech output with background noise and reverberation extracted from the original audio. We report on a subjective evaluation of automatic dubbing of excerpts of TED Talks from English into Italian, which measures the perceived naturalness of automatic dubbing and the relative importance of each proposed enhancement.


page 3

page 4


Machine Translation Verbosity Control for Automatic Dubbing

Automatic dubbing aims at seamlessly replacing the speech in a video doc...

Machine Assisted Analysis of Vowel Length Contrasts in Wolof

Growing digital archives and improving algorithms for automatic analysis...

Is 42 the Answer to Everything in Subtitling-oriented Speech Translation?

Subtitling is becoming increasingly important for disseminating informat...

Prosodic Alignment for off-screen automatic dubbing

The goal of automatic dubbing is to perform speech-to-speech translation...

Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing

We investigate how humans perform the task of dubbing video content from...

Puppet Dubbing

Dubbing puppet videos to make the characters (e.g. Kermit the Frog) conv...

Dialogue Enhancement in Object-based Audio – Evaluating the Benefit on People above 65

Due to age-related hearing loss, elderly people often struggle with foll...

Please sign up or login with your details

Forgot password? Click here to reset