Improving American Sign Language Recognition with Synthetic Data

05/21/2020
by   Jungi Kim, et al.
0

There is a need for real-time communication between the deaf and hearing without the aid of an interpreter. Developing a machine translation (MT) system between sign and spoken languages is a multimodal task since sign language is a visual language, which involves the automatic recognition and translation of video images. In this paper, we present the research we have been carrying out to build an automated sign language recognizer (ASLR), which is the core component of a machine translation (MT) system between American Sign Language (ASL) and English. Developing an ASLR is a challenging task due to the lack of sufficient quantities of annotated ASL-English parallel corpora for training, testing and developing an ASLR. This paper describes the research we have been conducting to explore a range of different techniques for automatically generating synthetic data from existing datasets to improve the accuracy of ASLR. This work involved experimentation with several algorithms with varying amounts of synthetic data and evaluations of their effectiveness. It was demonstrated that automatically creating valid synthetic training data through simple image manipulation of ASL video recordings improves the performance of the ASLR task.

READ FULL TEXT

page 4

page 7

page 8

page 9

research
10/11/2022

Machine Translation between Spoken Languages and Signed Languages Represented in SignWriting

This paper presents work on novel machine translation (MT) systems betwe...
research
08/14/2021

Findings of the LoResMT 2021 Shared Task on COVID and Sign Language for Low-resource Languages

We present the findings of the LoResMT 2021 shared task which focuses on...
research
05/02/2023

SLTUNET: A Simple Unified Model for Sign Language Translation

Despite recent successes with neural models for sign language translatio...
research
06/26/2023

Data-Driven Approach for Formality-Sensitive Machine Translation: Language-Specific Handling and Synthetic Data Generation

In this paper, we introduce a data-driven approach for Formality-Sensiti...
research
09/03/2020

Modeling Global Body Configurations in American Sign Language

American Sign Language (ASL) is the fourth most commonly used language i...
research
05/11/2023

The First Parallel Corpora for Kurdish Sign Language

Kurdish Sign Language (KuSL) is the natural language of the Kurdish Deaf...
research
05/06/2020

Shape of synth to come: Why we should use synthetic data for English surface realization

The Surface Realization Shared Tasks of 2018 and 2019 were Natural Langu...

Please sign up or login with your details

Forgot password? Click here to reset