Log In Sign Up

Fluent and Low-latency Simultaneous Speech-to-Speech Translation with Self-adaptive Training

by   Renjie Zheng, et al.

Simultaneous speech-to-speech translation is widely useful but extremely challenging, since it needs to generate target-language speech concurrently with the source-language speech, with only a few seconds delay. In addition, it needs to continuously translate a stream of sentences, but all recent solutions merely focus on the single-sentence scenario. As a result, current approaches accumulate latencies progressively when the speaker talks faster, and introduce unnatural pauses when the speaker talks slower. To overcome these issues, we propose Self-Adaptive Translation (SAT) which flexibly adjusts the length of translations to accommodate different source speech rates. At similar levels of translation quality (as measured by BLEU), our method generates more fluent target speech (as measured by the naturalness metric MOS) with substantially lower latency than the baseline, in both Zh <-> En directions.


page 1

page 2

page 3

page 4


STACL: Simultaneous Translation with Integrated Anticipation and Controllable Latency

Simultaneous translation, which translates sentences before they are fin...

Average Token Delay: A Latency Metric for Simultaneous Translation

Simultaneous translation is a task in which translation begins before th...

Re-Translation Strategies For Long Form, Simultaneous, Spoken Language Translation

We investigate the problem of simultaneous machine translation of long-f...

SimulEval: An Evaluation Toolkit for Simultaneous Translation

Simultaneous translation on both text and speech focuses on a real-time ...

Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translation

Simultaneous speech translation (SimulST) systems aim at generating thei...

Textless Speech-to-Speech Translation on Real Data

We present a textless speech-to-speech translation (S2ST) system that ca...

Simultaneous Speech Translation for Live Subtitling: from Delay to Display

With the increased audiovisualisation of communication, the need for liv...