DeepAI AI Chat
Log In Sign Up

Neural Sign Language Translation based on Human Keypoint Estimation

by   Sang-Ki Ko, et al.

We propose a sign language translation system based on human keypoint estimation. It is well-known that many problems in the field of computer vision require a massive amount of dataset to train deep neural network models. The situation is even worse when it comes to the sign language translation problem as it is far more difficult to collect high-quality training data. In this paper, we introduce the KETI sign language dataset which consists of 11,578 videos of high resolution and quality. Considering the fact that each country has a different and unique sign language, the KETI sign language dataset can be the starting line for further research on the Korean sign language translation. Using the KETI sign language dataset, we develop a neural network model for translating sign videos into natural language sentences by utilizing the human keypoints extracted from a face, hands, and body parts. The obtained human keypoint vector is normalized by the mean and standard deviation of the keypoints and used as input to our translation model based on the sequence-to-sequence architecture. As a result, we show that our approach is robust even when the size of the training data is not sufficient. Our translation model achieves 94.6 the validation set (test set, respectively) for 105 sentences that can be used in emergency situations. We compare several types of our neural sign translation models based on different attention mechanisms in terms of classical metrics for measuring the translation performance.


page 3

page 11

page 12

page 13


A two-way translation system of Chinese sign language based on computer vision

As the main means of communication for deaf people, sign language has a ...

Sequence-to-Sequence Natural Language to Humanoid Robot Sign Language

This paper presents a study on natural language to sign language transla...

Keypoint based Sign Language Translation without Glosses

Sign Language Translation (SLT) is a task that has not been studied rela...

Natural Language-Assisted Sign Language Recognition

Sign languages are visual languages which convey information by signers'...

ASL Recognition with Metric-Learning based Lightweight Network

In the past decades the set of human tasks that are solved by machines w...

Semantic Network Model for Sign Language Comprehension

In this study, the authors propose a computational cognitive model for s...

Dataset for a Neural Natural Language Interface for Databases (NNLIDB)

Progress in natural language interfaces to databases (NLIDB) has been sl...