Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?

11/27/2017
by   Kensho Hara, et al.
0

The purpose of this study is to determine whether current video datasets have sufficient data for training very deep convolutional neural networks (CNNs) with spatio-temporal three-dimensional (3D) kernels. Recently, the performance levels of 3D CNNs in the field of action recognition have improved significantly. However, to date, conventional research has only explored relatively shallow3Darchitectures. We examine the architectures of various 3D CNNs from relatively shallow to very deep ones on current video datasets. Based on the results of those experiments, the following conclusions could be obtained: (i) ResNet-18 training resulted in significant overfitting for UCF-101, HMDB-51, and ActivityNet but not for Kinetics. (ii) The Kinetics dataset has sufficient data for training of deep 3D CNNs, and enables training of up to 152 ResNets layers, interestingly similar to 2D ResNets on ImageNet. (iii) Kinetics pretrained simple 3D architectures outperforms complex2D architectures, and the pretrained ResNeXt-101 achieved 94.5 UCF-101 and HMDB-51, respectively. The use of 2D CNNs trained on ImageNet has produced significant progress in various tasks in image. We believe that using deep 3D CNNs together with Kinetics will retrace the successful history of 2D CNNs and ImageNet, and stimulate advances in computer vision for videos. The codes and pretrained models used in this study are publicly available.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/25/2017

Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition

Convolutional neural networks with spatio-temporal 3D kernels (3D CNNs) ...
research
04/04/2019

Resource Efficient 3D Convolutional Neural Networks

Recently, convolutional neural networks with 3D kernels (3D CNNs) have b...
research
10/22/2018

Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?

This paper seeks to answer the question: as the (near-) orthogonality of...
research
04/10/2020

Would Mega-scale Datasets Further Enhance Spatiotemporal 3D CNNs?

How can we collect and use a video dataset to further improve spatiotemp...
research
07/26/2018

A Better Baseline for AVA

We introduce a simple baseline for action localization on the AVA datase...
research
09/11/2019

Comparative Analysis of CNN-based Spatiotemporal Reasoning in Videos

Understanding actions and gestures in video streams requires temporal re...
research
07/23/2022

Intelligent 3D Network Protocol for Multimedia Data Classification using Deep Learning

In videos, the human's actions are of three-dimensional (3D) signals. Th...

Please sign up or login with your details

Forgot password? Click here to reset