DeepAI AI Chat
Log In Sign Up

Improving Sign Language Translation with Monolingual Data by Sign Back-Translation

by   Hao Zhou, et al.

Despite existing pioneering works on sign language translation (SLT), there is a non-trivial obstacle, i.e., the limited quantity of parallel sign-text data. To tackle this parallel data bottleneck, we propose a sign back-translation (SignBT) approach, which incorporates massive spoken language texts into SLT training. With a text-to-gloss translation model, we first back-translate the monolingual text to its gloss sequence. Then, the paired sign sequence is generated by splicing pieces from an estimated gloss-to-sign bank at the feature level. Finally, the synthetic parallel data serves as a strong supplement for the end-to-end training of the encoder-decoder SLT framework. To promote the SLT research, we further contribute CSL-Daily, a large-scale continuous SLT dataset. It provides both spoken language translations and gloss-level annotations. The topic revolves around people's daily lives (e.g., travel, shopping, medical care), the most likely SLT application scenario. Extensive experimental results and analysis of SLT methods are reported on CSL-Daily. With the proposed sign back-translation method, we obtain a substantial improvement over previous state-of-the-art SLT methods.


page 2

page 3

page 5

page 6

page 7

page 8

page 9

page 10


Scaling Back-Translation with Domain Text Generation for Sign Language Gloss Translation

Sign language gloss translation aims to translate the sign glosses into ...

A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation

This paper proposes a simple transfer learning baseline for sign languag...

Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation

Prior work on Sign Language Translation has shown that having a mid-leve...

SimulSLT: End-to-End Simultaneous Sign Language Translation

Sign language translation as a kind of technology with profound social s...

Example-Based Machine Translation from Text to a Hierarchical Representation of Sign Language

This article presents an original method for Text-to-Sign Translation. I...

DeepASL: Enabling Ubiquitous and Non-Intrusive Word and Sentence-Level Sign Language Translation

There is an undeniable communication barrier between deaf people and peo...

Keypoint based Sign Language Translation without Glosses

Sign Language Translation (SLT) is a task that has not been studied rela...