Watch, read and lookup: learning to spot signs from multiple supervisors

by   Liliane Momeni, et al.

The focus of this work is sign spotting - given a video of an isolated sign, our task is to identify whether and where it has been signed in a continuous, co-articulated sign language video. To achieve this sign spotting task, we train a model using multiple types of available supervision by: (1) watching existing sparsely labelled footage; (2) reading associated subtitles (readily available translations of the signed content) which provide additional weak-supervision; (3) looking up words (for which no co-articulated labelled examples are available) in visual sign language dictionaries to enable novel sign spotting. These three tasks are integrated into a unified learning framework using the principles of Noise Contrastive Estimation and Multiple Instance Learning. We validate the effectiveness of our approach on low-shot sign spotting benchmarks. In addition, we contribute a machine-readable British Sign Language (BSL) dictionary dataset of isolated signs, BSLDict, to facilitate study of this task. The dataset, models and code are available at our project page.


page 5

page 14

page 27


Scaling up sign spotting through sign language dictionaries

The focus of this work is sign spotting - given a video of an isolated s...

Word separation in continuous sign language using isolated signs and post-processing

Continuous Sign Language Recognition (CSLR) is a long challenging task i...

ASL Video Corpora Sign Bank: Resources Available through the American Sign Language Linguistic Research Project (ASLLRP)

The American Sign Language Linguistic Research Project (ASLLRP) provides...

AnonySIGN: Novel Human Appearance Synthesis for Sign Language Video Anonymisation

The visual anonymisation of sign language data is an essential task to a...

Sign Language Video Retrieval with Free-Form Textual Queries

Systems that can efficiently search collections of sign language videos ...

Automatic Hand Sign Recognition: Identify Unusuality through Latent Cognizance

Sign language is a main communication channel among hearing disability c...

American Sign Language fingerspelling recognition from video: Methods for unrestricted recognition and signer-independence

In this thesis, we study the problem of recognizing video sequences of f...

Code Repositories


Watch, read and lookup: learning to spot signs from multiple supervisors, ACCV 2020 (Best Application Paper)

view repo