Explore More Guidance: A Task-aware Instruction Network for Sign Language Translation Enhanced with Data Augmentation

04/12/2022
by   Yong Cao, et al.
4

Sign language recognition and translation first uses a recognition module to generate glosses from sign language videos and then employs a translation module to translate glosses into spoken sentences. Most existing works focus on the recognition step, while paying less attention to sign language translation. In this work, we propose a task-aware instruction network, namely TIN-SLT, for sign language translation, by introducing the instruction module and the learning-based feature fuse strategy into a Transformer network. In this way, the pre-trained model's language ability can be well explored and utilized to further boost the translation performance. Moreover, by exploring the representation space of sign language glosses and target spoken language, we propose a multi-level data augmentation scheme to adjust the data distribution of the training set. We conduct extensive experiments on two challenging benchmark datasets, PHOENIX-2014-T and ASLG-PC12, on which our method outperforms former best solutions by 1.65 and 1.42 in terms of BLEU-4. Our code is published at https://github.com/yongcaoplus/TIN-SLT.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/02/2022

Tackling Low-Resourced Sign Language Translation: UPC at WMT-SLT 22

This paper describes the system developed at the Universitat Politècnica...
research
03/30/2020

Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation

Prior work on Sign Language Translation has shown that having a mid-leve...
research
07/12/2022

Building Korean Sign Language Augmentation (KoSLA) Corpus with Data Augmentation Technique

We present an efficient framework of corpus for sign language translatio...
research
08/18/2023

Is context all you need? Scaling Neural Sign Language Translation to Large Domains of Discourse

Sign Language Translation (SLT) is a challenging task that aims to gener...
research
10/24/2022

Clean Text and Full-Body Transformer: Microsoft's Submission to the WMT22 Shared Task on Sign Language Translation

This paper describes Microsoft's submission to the first shared task on ...
research
04/22/2022

Keypoint based Sign Language Translation without Glosses

Sign Language Translation (SLT) is a task that has not been studied rela...
research
04/21/2023

Better Sign Language Translation with Monolingual Data

Sign language translation (SLT) systems, which are often decomposed into...

Please sign up or login with your details

Forgot password? Click here to reset