LIPSFUS: A neuromorphic dataset for audio-visual sensory fusion of lip reading

03/28/2023
by   Antonio Rios-Navarro, et al.
0

This paper presents a sensory fusion neuromorphic dataset collected with precise temporal synchronization using a set of Address-Event-Representation sensors and tools. The target application is the lip reading of several keywords for different machine learning applications, such as digits, robotic commands, and auxiliary rich phonetic short words. The dataset is enlarged with a spiking version of an audio-visual lip reading dataset collected with frame-based cameras. LIPSFUS is publicly available and it has been validated with a deep learning architecture for audio and visual classification. It is intended for sensory fusion architectures based on both artificial and spiking neural network algorithms.

READ FULL TEXT

page 2

page 3

research
10/25/2018

Adaptive motor control and learning in a spiking neural network realised on a mixed-signal neuromorphic processor

Neuromorphic computing is a new paradigm for design of both the computin...
research
04/28/2018

A Bimodal Learning Approach to Assist Multi-sensory Effects Synchronization

In mulsemedia applications, traditional media content (text, image, audi...
research
05/08/2020

ST-MNIST – The Spiking Tactile MNIST Neuromorphic Dataset

Tactile sensing is an essential modality for smart robots as it enables ...
research
10/23/2021

ES-ImageNet: A Million Event-Stream Classification Dataset for Spiking Neural Networks

With event-driven algorithms, especially the spiking neural networks (SN...
research
02/15/2018

Deep Learning for Lip Reading using Audio-Visual Information for Urdu Language

Human lip-reading is a challenging task. It requires not only knowledge ...
research
05/27/2022

Towards a Design Framework for TNN-Based Neuromorphic Sensory Processing Units

Temporal Neural Networks (TNNs) are spiking neural networks that exhibit...
research
12/27/2021

AU Dataset for Visuo-Haptic Object Recognition for Robots

Multimodal object recognition is still an emerging field. Thus, publicly...

Please sign up or login with your details

Forgot password? Click here to reset