Visual Alignment Constraint for Continuous Sign Language Recognition

04/06/2021
by   Yuecong Min, et al.
0

Vision-based Continuous Sign Language Recognition (CSLR) aims to recognize unsegmented gestures from image sequences. To better train CSLR models, the iterative training scheme is widely adopted to alleviate the overfitting of the alignment model. Although the iterative training scheme can improve performance, it will also increase the training time. In this work, we revisit the overfitting problem in recent CTC-based CSLR works and attribute it to the insufficient training of the feature extractor. To solve this problem, we propose a Visual Alignment Constraint (VAC) to enhance the feature extractor with more alignment supervision. Specifically, the proposed VAC is composed of two auxiliary losses: one makes predictions based on visual features only, and the other aligns short-term visual and long-term contextual features. Moreover, we further propose two metrics to evaluate the contributions of the feature extractor and the alignment model, which provide evidence for the overfitting problem. The proposed VAC achieves competitive performance on two challenging CSLR datasets and experimental results show its effectiveness.

READ FULL TEXT
research
12/26/2022

Improving Continuous Sign Language Recognition with Consistency Constraints and Signer Removal

Most deep-learning-based continuous sign language recognition (CSLR) mod...
research
08/04/2019

SF-Net: Structured Feature Network for Continuous Sign Language Recognition

Continuous sign language recognition (SLR) aims to translate a signing s...
research
09/01/2022

Topic Detection in Continuous Sign Language Videos

Significant progress has been made recently on challenging tasks in auto...
research
03/27/2023

AIR-DA: Adversarial Image Reconstruction for Unsupervised Domain Adaptive Object Detection

Unsupervised domain adaptive object detection is a challenging vision ta...
research
03/28/2018

Exploiting Recurrent Neural Networks and Leap Motion Controller for Sign Language and Semaphoric Gesture Recognition

In human interactions, hands are a powerful way of expressing informatio...
research
06/19/2020

Evaluation Of Hidden Markov Models Using Deep CNN Features In Isolated Sign Recognition

Isolated sign recognition from video streams is a challenging problem du...

Please sign up or login with your details

Forgot password? Click here to reset