DeepAI AI Chat
Log In Sign Up

A Fine-Grained Visual Attention Approach for Fingerspelling Recognition in the Wild

by   Kamala Gajurel, et al.

Fingerspelling in sign language has been the means of communicating technical terms and proper nouns when they do not have dedicated sign language gestures. Automatic recognition of fingerspelling can help resolve communication barriers when interacting with deaf people. The main challenges prevalent in fingerspelling recognition are the ambiguity in the gestures and strong articulation of the hands. The automatic recognition model should address high inter-class visual similarity and high intra-class variation in the gestures. Most of the existing research in fingerspelling recognition has focused on the dataset collected in a controlled environment. The recent collection of a large-scale annotated fingerspelling dataset in the wild, from social media and online platforms, captures the challenges in a real-world scenario. In this work, we propose a fine-grained visual attention mechanism using the Transformer model for the sequence-to-sequence prediction task in the wild dataset. The fine-grained attention is achieved by utilizing the change in motion of the video frames (optical flow) in sequential context-based attention along with a Transformer encoder model. The unsegmented continuous video dataset is jointly trained by balancing the Connectionist Temporal Classification (CTC) loss and the maximum-entropy loss. The proposed approach can capture better fine-grained attention in a single iteration. Experiment evaluations show that it outperforms the state-of-the-art approaches.


Fingerspelling recognition in the wild with iterative visual attention

Sign language recognition is a challenging gesture sequence recognition ...

The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition

Current approaches for fine-grained recognition do the following: First,...

Regional Attention Network (RAN) for Head Pose and Fine-grained Gesture Recognition

Affect is often expressed via non-verbal body language such as actions/g...

American Sign Language fingerspelling recognition in the wild

We address the problem of American Sign Language fingerspelling recognit...

Global-local Enhancement Network for NMFs-aware Sign Language Recognition

Sign language recognition (SLR) is a challenging problem, involving comp...

Fine-Grained Visual Recognition with Batch Confusion Norm

We introduce a regularization concept based on the proposed Batch Confus...

Reconstructing Signing Avatars From Video Using Linguistic Priors

Sign language (SL) is the primary method of communication for the 70 mil...