DeepAI AI Chat
Log In Sign Up

A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation

by   Yutong Chen, et al.

This paper proposes a simple transfer learning baseline for sign language translation. Existing sign language datasets (e.g. PHOENIX-2014T, CSL-Daily) contain only about 10K-20K pairs of sign videos, gloss annotations and texts, which are an order of magnitude smaller than typical parallel data for training spoken language translation models. Data is thus a bottleneck for training effective sign language translation models. To mitigate this problem, we propose to progressively pretrain the model from general-domain datasets that include a large amount of external supervision to within-domain datasets. Concretely, we pretrain the sign-to-gloss visual network on the general domain of human actions and the within-domain of a sign-to-gloss dataset, and pretrain the gloss-to-text translation network on the general domain of a multilingual corpus and the within-domain of a gloss-to-text corpus. The joint model is fine-tuned with an additional module named the visual-language mapper that connects the two networks. This simple baseline surpasses the previous state-of-the-art results on two sign language translation benchmarks, demonstrating the effectiveness of transfer learning. With its simplicity and strong performance, this approach can serve as a solid baseline for future research.


page 1

page 2

page 3

page 4


Scaling Back-Translation with Domain Text Generation for Sign Language Gloss Translation

Sign language gloss translation aims to translate the sign glosses into ...

Improving Sign Language Translation with Monolingual Data by Sign Back-Translation

Despite existing pioneering works on sign language translation (SLT), th...

Transfer Learning for British Sign Language Modelling

Automatic speech recognition and spoken dialogue systems have made great...

Two-Stream Network for Sign Language Recognition and Translation

Sign languages are visual languages using manual articulations and non-m...

Neural Sign Language Translation by Learning Tokenization

Sign Language Translation has attained considerable success recently, ra...

Master Thesis: Neural Sign Language Translation by Learning Tokenization

In this thesis, we propose a multitask learning based method to improve ...

Building Korean Sign Language Augmentation (KoSLA) Corpus with Data Augmentation Technique

We present an efficient framework of corpus for sign language translatio...