Combining Efficient and Precise Sign Language Recognition: Good pose estimation library is all you need

by   Matyáš Boháček, et al.

Sign language recognition could significantly improve the user experience for d/Deaf people with the general consumer technology, such as IoT devices or videoconferencing. However, current sign language recognition architectures are usually computationally heavy and require robust GPU-equipped hardware to run in real-time. Some models aim for lower-end devices (such as smartphones) by minimizing their size and complexity, which leads to worse accuracy. This highly scrutinizes accurate in-the-wild applications. We build upon the SPOTER architecture, which belongs to the latter group of light methods, as it came close to the performance of large models employed for this task. By substituting its original third-party pose estimation module with the MediaPipe library, we achieve an overall state-of-the-art result on the WLASL100 dataset. Significantly, our method beats previous larger architectures while still being twice as computationally efficient and almost 11 times faster on inference when compared to a relevant benchmark. To demonstrate our method's combined efficiency and precision, we built an online demo that enables users to translate sign lemmas of American sign language in their browsers. This is the first publicly available online application demonstrating this task to the best of our knowledge.


page 2

page 3


Real-Time Sign Language Detection using Human Pose Estimation

We propose a lightweight real-time sign language detection model, as we ...

Evaluating the Immediate Applicability of Pose Estimation for Sign Language Recognition

Signed languages are visual languages produced by the movement of the ha...

Bangla sign language recognition using concatenated BdSL network

Sign language is the only medium of communication for the hearing impair...

Learning from What is Already Out There: Few-shot Sign Language Recognition with Online Dictionaries

Today's sign language recognition models require large training corpora ...

American Sign Language Alphabet Recognition using Deep Learning

Tremendous headway has been made in the field of 3D hand pose estimation...

Fingerspelling Detection in American Sign Language

Fingerspelling, in which words are signed letter by letter, is an import...

Multi-View Bangla Sign Language(MV-BSL) Dataset and Continuous BSL Recognition

Being able to express our thoughts, feelings, and ideas to one another i...

Please sign up or login with your details

Forgot password? Click here to reset