Improving Isochronous Machine Translation with Target Factors and Auxiliary Counters

05/22/2023
by   Proyag Pal, et al.
0

To translate speech for automatic dubbing, machine translation needs to be isochronous, i.e. translated speech needs to be aligned with the source in terms of speech durations. We introduce target factors in a transformer model to predict durations jointly with target language phoneme sequences. We also introduce auxiliary counters to help the decoder to keep track of the timing information while generating target phonemes. We show that our model improves translation quality and isochrony compared to previous work where the translation model is instead trained to predict interleaved sequences of phonemes and durations.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/25/2023

Jointly Optimizing Translations and Speech Timing to Improve Isochrony in Automatic Dubbing

Automatic dubbing (AD) is the task of translating the original speech in...
research
04/05/2020

Reference Language based Unsupervised Neural Machine Translation

Exploiting common language as an auxiliary for better translation has a ...
research
12/16/2021

Prosody-Aware Neural Machine Translation for Dubbing

We introduce the task of prosody-aware machine translation which aims at...
research
07/08/2019

Correct-and-Memorize: Learning to Translate from Interactive Revisions

State-of-the-art machine translation models are still not on par with hu...
research
11/07/2018

Towards Fluent Translations from Disfluent Speech

When translating from speech, special consideration for conversational s...
research
07/05/2016

Target-Side Context for Discriminative Models in Statistical Machine Translation

Discriminative translation models utilizing source context have been sho...
research
05/31/2022

VALHALLA: Visual Hallucination for Machine Translation

Designing better machine translation systems by considering auxiliary in...

Please sign up or login with your details

Forgot password? Click here to reset