Evaluation Of Hidden Markov Models Using Deep CNN Features In Isolated Sign Recognition

by   Anil Osman Tur, et al.

Isolated sign recognition from video streams is a challenging problem due to the multi-modal nature of the signs, where both local and global hand features and face gestures needs to be attended simultaneously. This problem has recently been studied widely using deep Convolutional Neural Network (CNN) based features and Long Short-Term Memory (LSTM) based deep sequence models. However, the current literature is lack of providing empirical analysis using Hidden Markov Models (HMMs) with deep features. In this study, we provide a framework that is composed of three modules to solve isolated sign recognition problem using different sequence models. The dimensions of deep features are usually too large to work with HMM models. To solve this problem, we propose two alternative CNN based architectures as the second module in our framework, to reduce deep feature dimensions effectively. After extensive experiments, we show that using pretrained Resnet50 features and one of our CNN based dimension reduction models, HMMs can classify isolated signs with 90.15% accuracy in Montalbano dataset using RGB and Skeletal data. This performance is comparable with the current LSTM based models. HMMs have fewer parameters and can be trained and run on commodity computers fast, without requiring GPUs. Therefore, our analysis with deep features show that HMMs could also be utilized as well as deep sequence models in challenging isolated sign recognition problem.


page 5

page 6

page 8


Using Motion History Images with 3D Convolutional Networks in Isolated Sign Language Recognition

Sign language recognition using computational models is a challenging pr...

AUTSL: A Large Scale Multi-modal Turkish Sign Language Dataset and Baseline Methods

Sign language recognition is a challenging problem where signs are ident...

Egyptian Sign Language Recognition Using CNN and LSTM

Sign language is a set of gestures that deaf people use to communicate. ...

Global-local Enhancement Network for NMFs-aware Sign Language Recognition

Sign language recognition (SLR) is a challenging problem, involving comp...

ChaLearn LAP Large Scale Signer Independent Isolated Sign Language Recognition Challenge: Design, Results and Future Research

The performances of Sign Language Recognition (SLR) systems have improve...

American Sign Language fingerspelling recognition from video: Methods for unrestricted recognition and signer-independence

In this thesis, we study the problem of recognizing video sequences of f...

Visual Alignment Constraint for Continuous Sign Language Recognition

Vision-based Continuous Sign Language Recognition (CSLR) aims to recogni...

Please sign up or login with your details

Forgot password? Click here to reset