Video-based Sign Language Recognition without Temporal Segmentation

01/30/2018
by   Jie Huang, et al.
0

Millions of hearing impaired people around the world routinely use some variants of sign languages to communicate, thus the automatic translation of a sign language is meaningful and important. Currently, there are two sub-problems in Sign Language Recognition (SLR), i.e., isolated SLR that recognizes word by word and continuous SLR that translates entire sentences. Existing continuous SLR methods typically utilize isolated SLRs as building blocks, with an extra layer of preprocessing (temporal segmentation) and another layer of post-processing (sentence synthesis). Unfortunately, temporal segmentation itself is non-trivial and inevitably propagates errors into subsequent steps. Worse still, isolated SLR methods typically require strenuous labeling of each word separately in a sentence, severely limiting the amount of attainable training data. To address these challenges, we propose a novel continuous sign recognition framework, the Hierarchical Attention Network with Latent Space (LS-HAN), which eliminates the preprocessing of temporal segmentation. The proposed LS-HAN consists of three components: a two-stream Convolutional Neural Network (CNN) for video feature representation generation, a Latent Space (LS) for semantic gap bridging, and a Hierarchical Attention Network (HAN) for latent space based recognition. Experiments are carried out on two large scale datasets. Experimental results demonstrate the effectiveness of the proposed framework.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/02/2022

Word separation in continuous sign language using isolated signs and post-processing

Continuous Sign Language Recognition (CSLR) is a long challenging task i...
research
06/03/2023

A two-way translation system of Chinese sign language based on computer vision

As the main means of communication for deaf people, sign language has a ...
research
10/12/2020

TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation

Sign language translation (SLT) aims to interpret sign video sequences i...
research
11/25/2020

Sign language segmentation with temporal convolutional networks

The objective of this work is to determine the location of temporal boun...
research
03/08/2020

Transferring Cross-domain Knowledge for Video Sign Language Recognition

Word-level sign language recognition (WSLR) is a fundamental task in sig...
research
02/21/2018

DeepASL: Enabling Ubiquitous and Non-Intrusive Word and Sentence-Level Sign Language Translation

There is an undeniable communication barrier between deaf people and peo...
research
08/04/2019

SF-Net: Structured Feature Network for Continuous Sign Language Recognition

Continuous sign language recognition (SLR) aims to translate a signing s...

Please sign up or login with your details

Forgot password? Click here to reset