DeepAI AI Chat
Log In Sign Up

Two-Stream Network for Sign Language Recognition and Translation

by   Yutong Chen, et al.

Sign languages are visual languages using manual articulations and non-manual elements to convey information. For sign language recognition and translation, the majority of existing approaches directly encode RGB videos into hidden representations. RGB videos, however, are raw signals with substantial visual redundancy, leading the encoder to overlook the key information for sign language understanding. To mitigate this problem and better incorporate domain knowledge, such as handshape and body movement, we introduce a dual visual encoder containing two separate streams to model both the raw videos and the keypoint sequences generated by an off-the-shelf keypoint estimator. To make the two streams interact with each other, we explore a variety of techniques, including bidirectional lateral connection, sign pyramid network with auxiliary supervision, and frame-level self-distillation. The resulting model is called TwoStream-SLR, which is competent for sign language recognition (SLR). TwoStream-SLR is extended to a sign language translation (SLT) model, TwoStream-SLT, by simply attaching an extra translation network. Experimentally, our TwoStream-SLR and TwoStream-SLT achieve state-of-the-art performance on SLR and SLT tasks across a series of datasets including Phoenix-2014, Phoenix-2014T, and CSL-Daily.


Natural Language-Assisted Sign Language Recognition

Sign languages are visual languages which convey information by signers'...

Keypoint based Sign Language Translation without Glosses

Sign Language Translation (SLT) is a task that has not been studied rela...

A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation

This paper proposes a simple transfer learning baseline for sign languag...

Neural Sign Language Translation based on Human Keypoint Estimation

We propose a sign language translation system based on human keypoint es...

ZS-SLR: Zero-Shot Sign Language Recognition from RGB-D Videos

Sign Language Recognition (SLR) is a challenging research area in comput...

Improving Continuous Sign Language Recognition with Consistency Constraints and Signer Removal

Most deep-learning-based continuous sign language recognition (CSLR) mod...

Aligning Subtitles in Sign Language Videos

The goal of this work is to temporally align asynchronous subtitles in s...