Watch, read and lookup: learning to spot signs from multiple supervisors

10/08/2020
by   Liliane Momeni, et al.
8

The focus of this work is sign spotting - given a video of an isolated sign, our task is to identify whether and where it has been signed in a continuous, co-articulated sign language video. To achieve this sign spotting task, we train a model using multiple types of available supervision by: (1) watching existing sparsely labelled footage; (2) reading associated subtitles (readily available translations of the signed content) which provide additional weak-supervision; (3) looking up words (for which no co-articulated labelled examples are available) in visual sign language dictionaries to enable novel sign spotting. These three tasks are integrated into a unified learning framework using the principles of Noise Contrastive Estimation and Multiple Instance Learning. We validate the effectiveness of our approach on low-shot sign spotting benchmarks. In addition, we contribute a machine-readable British Sign Language (BSL) dictionary dataset of isolated signs, BSLDict, to facilitate study of this task. The dataset, models and code are available at our project page.

READ FULL TEXT

page 5

page 14

page 27

research
05/09/2022

Scaling up sign spotting through sign language dictionaries

The focus of this work is sign spotting - given a video of an isolated s...
research
04/12/2023

ASL Citizen: A Community-Sourced Dataset for Advancing Isolated Sign Language Recognition

Sign languages are used as a primary language by approximately 70 millio...
research
04/02/2022

Word separation in continuous sign language using isolated signs and post-processing

Continuous Sign Language Recognition (CSLR) is a long challenging task i...
research
08/21/2023

Improving Continuous Sign Language Recognition with Cross-Lingual Signs

This work dedicates to continuous sign language recognition (CSLR), whic...
research
01/19/2022

ASL Video Corpora Sign Bank: Resources Available through the American Sign Language Linguistic Research Project (ASLLRP)

The American Sign Language Linguistic Research Project (ASLLRP) provides...
research
05/23/2023

Slovo: Russian Sign Language Dataset

One of the main challenges of the sign language recognition task is the ...
research
10/29/2021

Automatic Hand Sign Recognition: Identify Unusuality through Latent Cognizance

Sign language is a main communication channel among hearing disability c...

Please sign up or login with your details

Forgot password? Click here to reset