Scaling up sign spotting through sign language dictionaries

05/09/2022
by   Gul Varol, et al.
1

The focus of this work is sign spotting - given a video of an isolated sign, our task is to identify whether and where it has been signed in a continuous, co-articulated sign language video. To achieve this sign spotting task, we train a model using multiple types of available supervision by: (1) watching existing footage which is sparsely labelled using mouthing cues; (2) reading associated subtitles (readily available translations of the signed content) which provide additional weak-supervision; (3) looking up words (for which no co-articulated labelled examples are available) in visual sign language dictionaries to enable novel sign spotting. These three tasks are integrated into a unified learning framework using the principles of Noise Contrastive Estimation and Multiple Instance Learning. We validate the effectiveness of our approach on low-shot sign spotting benchmarks. In addition, we contribute a machine-readable British Sign Language (BSL) dictionary dataset of isolated signs, BSLDict, to facilitate study of this task. The dataset, models and code are available at our project page.

READ FULL TEXT

page 4

page 5

page 9

page 12

page 13

page 16

page 23

page 24

10/08/2020

Watch, read and lookup: learning to spot signs from multiple supervisors

The focus of this work is sign spotting - given a video of an isolated s...
11/05/2021

BBC-Oxford British Sign Language Dataset

In this work, we introduce the BBC-Oxford British Sign Language (BOBSL) ...
04/02/2022

Word separation in continuous sign language using isolated signs and post-processing

Continuous Sign Language Recognition (CSLR) is a long challenging task i...
04/12/2023

ASL Citizen: A Community-Sourced Dataset for Advancing Isolated Sign Language Recognition

Sign languages are used as a primary language by approximately 70 millio...
01/10/2023

Learning from What is Already Out There: Few-shot Sign Language Recognition with Online Dictionaries

Today's sign language recognition models require large training corpora ...
04/28/2021

Sign Segmentation with Changepoint-Modulated Pseudo-Labelling

The objective of this work is to find temporal boundaries between signs ...

Please sign up or login with your details

Forgot password? Click here to reset