DeepAI AI Chat
Log In Sign Up

Depthwise Spatio-Temporal STFT Convolutional Neural Networks for Human Action Recognition

07/22/2020
by   Sudhakar Kumawat, et al.
5

Conventional 3D convolutional neural networks (CNNs) are computationally expensive, memory intensive, prone to overfitting, and most importantly, there is a need to improve their feature learning capabilities. To address these issues, we propose spatio-temporal short term Fourier transform (STFT) blocks, a new class of convolutional blocks that can serve as an alternative to the 3D convolutional layer and its variants in 3D CNNs. An STFT block consists of non-trainable convolution layers that capture spatially and/or temporally local Fourier information using a STFT kernel at multiple low frequency points, followed by a set of trainable linear weights for learning channel correlations. The STFT blocks significantly reduce the space-time complexity in 3D CNNs. In general, they use 3.5 to 4.5 times less parameters and 1.5 to 1.8 times less computational costs when compared to the state-of-the-art methods. Furthermore, their feature learning capabilities are significantly better than the conventional 3D convolutional layer and its variants. Our extensive evaluation on seven action recognition datasets, including Something-something v1 and v2, Jester, Diving-48, Kinetics-400, UCF 101, and HMDB 51, demonstrate that STFT blocks based 3D CNNs achieve on par or even better performance compared to the state-of-the-art methods.

READ FULL TEXT

page 2

page 4

page 6

page 7

04/06/2019

LP-3DCNN: Unveiling Local Phase in 3D Convolutional Neural Networks

Traditional 3D Convolutional Neural Networks (CNNs) are computationally ...
09/07/2019

Exploring Temporal Differences in 3D Convolutional Neural Networks

Traditional 3D convolutions are computationally expensive, memory intens...
11/08/2020

Right on Time: Multi-Temporal Convolutions for Human Action Recognition in Videos

The variations in the temporal performance of human actions observed in ...
01/27/2020

Depthwise-STFT based separable Convolutional Neural Networks

In this paper, we propose a new convolutional layer called Depthwise-STF...
11/10/2018

Skeleton-Based Action Recognition with Synchronous Local and Non-local Spatio-temporal Learning and Frequency Attention

Benefiting from its succinctness and robustness, skeleton-based human ac...
02/26/2019

STAR-Net: Action Recognition using Spatio-Temporal Activation Reprojection

While depth cameras and inertial sensors have been frequently leveraged ...
05/09/2017

Deep Spatio-temporal Manifold Network for Action Recognition

Visual data such as videos are often sampled from complex manifold. We p...