Watch, read and lookup: learning to spot signs from multiple supervisors

10/08/2020
by   Liliane Momeni, et al.
8

The focus of this work is sign spotting - given a video of an isolated sign, our task is to identify whether and where it has been signed in a continuous, co-articulated sign language video. To achieve this sign spotting task, we train a model using multiple types of available supervision by: (1) watching existing sparsely labelled footage; (2) reading associated subtitles (readily available translations of the signed content) which provide additional weak-supervision; (3) looking up words (for which no co-articulated labelled examples are available) in visual sign language dictionaries to enable novel sign spotting. These three tasks are integrated into a unified learning framework using the principles of Noise Contrastive Estimation and Multiple Instance Learning. We validate the effectiveness of our approach on low-shot sign spotting benchmarks. In addition, we contribute a machine-readable British Sign Language (BSL) dictionary dataset of isolated signs, BSLDict, to facilitate study of this task. The dataset, models and code are available at our project page.

READ FULL TEXT

page 5

page 14

page 27

05/09/2022

Scaling up sign spotting through sign language dictionaries

The focus of this work is sign spotting - given a video of an isolated s...
04/02/2022

Word separation in continuous sign language using isolated signs and post-processing

Continuous Sign Language Recognition (CSLR) is a long challenging task i...
01/19/2022

ASL Video Corpora Sign Bank: Resources Available through the American Sign Language Linguistic Research Project (ASLLRP)

The American Sign Language Linguistic Research Project (ASLLRP) provides...
07/22/2021

AnonySIGN: Novel Human Appearance Synthesis for Sign Language Video Anonymisation

The visual anonymisation of sign language data is an essential task to a...
01/07/2022

Sign Language Video Retrieval with Free-Form Textual Queries

Systems that can efficiently search collections of sign language videos ...
10/29/2021

Automatic Hand Sign Recognition: Identify Unusuality through Latent Cognizance

Sign language is a main communication channel among hearing disability c...
08/30/2016

American Sign Language fingerspelling recognition from video: Methods for unrestricted recognition and signer-independence

In this thesis, we study the problem of recognizing video sequences of f...

Code Repositories

bsldict

Watch, read and lookup: learning to spot signs from multiple supervisors, ACCV 2020 (Best Application Paper)


view repo