DeepAI AI Chat
Log In Sign Up

Keypoint based Sign Language Translation without Glosses

by   Youngmin Kim, et al.
Chung-Ang University

Sign Language Translation (SLT) is a task that has not been studied relatively much compared to the study of Sign Language Recognition (SLR). However, the SLR is a study that recognizes the unique grammar of sign language, which is different from the spoken language and has a problem that non-disabled people cannot easily interpret. So, we're going to solve the problem of translating directly spoken language in sign language video. To this end, we propose a new keypoint normalization method for performing translation based on the skeleton point of the signer and robustly normalizing these points in sign language translation. It contributed to performance improvement by a customized normalization method depending on the body parts. In addition, we propose a stochastic frame selection method that enables frame augmentation and sampling at the same time. Finally, it is translated into the spoken language through an Attention-based translation model. Our method can be applied to various datasets in a way that can be applied to datasets without glosses. In addition, quantitative experimental evaluation proved the excellence of our method.


page 1

page 2

page 3

page 4


Everybody Sign Now: Translating Spoken Language to Photo Realistic Sign Language Video

To be truly understandable and accepted by Deaf communities, an automati...

Neural Sign Language Translation based on Human Keypoint Estimation

We propose a sign language translation system based on human keypoint es...

Two-Stream Network for Sign Language Recognition and Translation

Sign languages are visual languages using manual articulations and non-m...

Modeling Global Body Configurations in American Sign Language

American Sign Language (ASL) is the fourth most commonly used language i...

Scaling Back-Translation with Domain Text Generation for Sign Language Gloss Translation

Sign language gloss translation aims to translate the sign glosses into ...

Explore More Guidance: A Task-aware Instruction Network for Sign Language Translation Enhanced with Data Augmentation

Sign language recognition and translation first uses a recognition modul...

Jointly Harnessing Prior Structures and Temporal Consistency for Sign Language Video Generation

Sign language is the window for people differently-abled to express thei...

Code Repositories


Sign Language Translation(BOAZ 15th Conference)

view repo