The VVAD-LRS3 Dataset for Visual Voice Activity Detection

09/28/2021
by   Adrian Lubitz, et al.
1

Robots are becoming everyday devices, increasing their interaction with humans. To make human-machine interaction more natural, cognitive features like Visual Voice Activity Detection (VVAD), which can detect whether a person is speaking or not, given visual input of a camera, need to be implemented. Neural networks are state of the art for tasks in Image Processing, Time Series Prediction, Natural Language Processing and other domains. Those Networks require large quantities of labeled data. Currently there are not many datasets for the task of VVAD. In this work we created a large scale dataset called the VVAD-LRS3 dataset, derived by automatic annotations from the LRS3 dataset. The VVAD-LRS3 dataset contains over 44K samples, over three times the next competitive dataset (WildVVAD). We evaluate different baselines on four kinds of features: facial and lip images, and facial and lip landmark features. With a Convolutional Neural Network Long Short Term Memory (CNN LSTM) on facial images an accuracy of 92 showed that they reach an accuracy of 87.93

READ FULL TEXT

page 2

page 4

page 5

page 6

research
02/25/2021

IMAGETOTEXT: IMAGE CAPTION GENERATION USING HYBRID RECURRENT NEURAL NETWORK

Generating a natural language description from images is an important pr...
research
05/04/2017

A Deep Learning Perspective on the Origin of Facial Expressions

Facial expressions play a significant role in human communication and be...
research
04/14/2021

CelebHair: A New Large-Scale Dataset for Hairstyle Recommendation based on CelebA

In this paper, we present a new large-scale dataset for hairstyle recomm...
research
05/10/2023

Mispronunciation Detection of Basic Quranic Recitation Rules using Deep Learning

In Islam, readers must apply a set of pronunciation rules called Tajweed...
research
09/23/2020

Learning Visual Voice Activity Detection with an Automatically Annotated Dataset

Visual voice activity detection (V-VAD) uses visual features to predict ...
research
07/21/2017

Multichannel Attention Network for Analyzing Visual Behavior in Public Speaking

Public speaking is an important aspect of human communication and intera...

Please sign up or login with your details

Forgot password? Click here to reset