SpotFast Networks with Memory Augmented Lateral Transformers for Lipreading

This paper presents a novel deep learning architecture for word-level lipreading. Previous works suggest a potential for incorporating a pretrained deep 3D Convolutional Neural Networks as a front-end feature extractor. We introduce a SpotFast networks, a variant of the state-of-the-art SlowFast networks for action recognition, which utilizes a temporal window as a spot pathway and all frames as a fast pathway. We further incorporate memory augmented lateral transformers to learn sequential features for classification. We evaluate the proposed model on the LRW dataset. The experiments show that our proposed model outperforms various state-of-the-art models and incorporating the memory augmented lateral transformers makes a 3.7 improvement to the SpotFast networks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/01/2021

VideoLightFormer: Lightweight Action Recognition using Transformers

Efficient video action recognition remains a challenging problem. One la...
research
12/16/2022

Dense Feature Memory Augmented Transformers for COVID-19 Vaccination Search Classification

With the devastating outbreak of COVID-19, vaccines are one of the cruci...
research
01/16/2019

Memory Augmented Deep Generative models for Forecasting the Next Shot Location in Tennis

This paper presents a novel framework for predicting shot location and t...
research
03/01/2022

Tricks and Plugins to GBM on Images and Sequences

Convolutional neural networks (CNNs) and transformers, which are compose...
research
05/31/2018

Lip Reading Using Convolutional Auto Encoders as Feature Extractor

Visual recognition of speech using the lip movement is called Lip-readin...
research
07/30/2019

EdgeNet: A novel approach for Arabic numeral classification

Despite the importance of handwritten numeral classification, a robust a...
research
06/18/2019

Crop Lodging Prediction from UAV-Acquired Images of Wheat and Canola using a DCNN Augmented with Handcrafted Texture Features

Lodging, the permanent bending over of food crops, leads to poor plant g...

Please sign up or login with your details

Forgot password? Click here to reset