Signing Outside the Studio: Benchmarking Background Robustness for Continuous Sign Language Recognition

by   Youngjoon Jang, et al.

The goal of this work is background-robust continuous sign language recognition. Most existing Continuous Sign Language Recognition (CSLR) benchmarks have fixed backgrounds and are filmed in studios with a static monochromatic background. However, signing is not limited only to studios in the real world. In order to analyze the robustness of CSLR models under background shifts, we first evaluate existing state-of-the-art CSLR models on diverse backgrounds. To synthesize the sign videos with a variety of backgrounds, we propose a pipeline to automatically generate a benchmark dataset utilizing existing CSLR benchmarks. Our newly constructed benchmark dataset consists of diverse scenes to simulate a real-world environment. We observe even the most recent CSLR method cannot recognize glosses well on our new dataset with changed backgrounds. In this regard, we also propose a simple yet effective training scheme including (1) background randomization and (2) feature disentanglement for CSLR models. The experimental results on our dataset demonstrate that our method generalizes well to other unseen background data with minimal additional training images.


page 2

page 4

page 5

page 10

page 18

page 19

page 22

page 23


Read and Attend: Temporal Localisation in Sign Language Videos

The objective of this work is to annotate sign instances across a broad ...

Learning From Context-Agnostic Synthetic Data

We present a new approach for synthesizing training data given only a si...

CURE-TSR: Challenging Unreal and Real Environments for Traffic Sign Recognition

In this paper, we investigate the robustness of traffic sign recognition...

Revisiting Shadow Detection: A New Benchmark Dataset for Complex World

Shadow detection in general photos is a nontrivial problem, due to the c...

Zero-Shot Sign Language Recognition: Can Textual Data Uncover Sign Languages?

We introduce the problem of zero-shot sign language recognition (ZSSLR),...

Quantitative Survey of the State of the Art in Sign Language Recognition

This work presents a meta study covering around 300 published sign langu...

ArabSign: A Multi-modality Dataset and Benchmark for Continuous Arabic Sign Language Recognition

Sign language recognition has attracted the interest of researchers in r...

Please sign up or login with your details

Forgot password? Click here to reset