Scaling up sign spotting through sign language dictionaries

by   Gul Varol, et al.

The focus of this work is sign spotting - given a video of an isolated sign, our task is to identify whether and where it has been signed in a continuous, co-articulated sign language video. To achieve this sign spotting task, we train a model using multiple types of available supervision by: (1) watching existing footage which is sparsely labelled using mouthing cues; (2) reading associated subtitles (readily available translations of the signed content) which provide additional weak-supervision; (3) looking up words (for which no co-articulated labelled examples are available) in visual sign language dictionaries to enable novel sign spotting. These three tasks are integrated into a unified learning framework using the principles of Noise Contrastive Estimation and Multiple Instance Learning. We validate the effectiveness of our approach on low-shot sign spotting benchmarks. In addition, we contribute a machine-readable British Sign Language (BSL) dictionary dataset of isolated signs, BSLDict, to facilitate study of this task. The dataset, models and code are available at our project page.


page 4

page 5

page 9

page 12

page 13

page 16

page 23

page 24


Watch, read and lookup: learning to spot signs from multiple supervisors

The focus of this work is sign spotting - given a video of an isolated s...

BBC-Oxford British Sign Language Dataset

In this work, we introduce the BBC-Oxford British Sign Language (BOBSL) ...

Word separation in continuous sign language using isolated signs and post-processing

Continuous Sign Language Recognition (CSLR) is a long challenging task i...

ASL Citizen: A Community-Sourced Dataset for Advancing Isolated Sign Language Recognition

Sign languages are used as a primary language by approximately 70 millio...

Learning from What is Already Out There: Few-shot Sign Language Recognition with Online Dictionaries

Today's sign language recognition models require large training corpora ...

Sign Segmentation with Changepoint-Modulated Pseudo-Labelling

The objective of this work is to find temporal boundaries between signs ...

Please sign up or login with your details

Forgot password? Click here to reset