Memory-Augmented Temporal Dynamic Learning for Action Recognition

04/30/2019
by   Yuan Yuan, et al.
0

Human actions captured in video sequences contain two crucial factors for action recognition, i.e., visual appearance and motion dynamics. To model these two aspects, Convolutional and Recurrent Neural Networks (CNNs and RNNs) are adopted in most existing successful methods for recognizing actions. However, CNN based methods are limited in modeling long-term motion dynamics. RNNs are able to learn temporal motion dynamics but lack effective ways to tackle unsteady dynamics in long-duration motion. In this work, we propose a memory-augmented temporal dynamic learning network, which learns to write the most evident information into an external memory module and ignore irrelevant ones. In particular, we present a differential memory controller to make a discrete decision on whether the external memory module should be updated with current feature. The discrete memory controller takes in the memory history, context embedding and current feature as inputs and controls information flow into the external memory module. Additionally, we train this discrete memory controller using straight-through estimator. We evaluate this end-to-end system on benchmark datasets (UCF101 and HMDB51) of human action recognition. The experimental results show consistent improvements on both datasets over prior works and our baselines.

READ FULL TEXT
research
08/13/2017

Lattice Long Short-Term Memory for Human Action Recognition

Human actions captured in video sequences are three-dimensional signals ...
research
02/26/2019

IF-TTN: Information Fused Temporal Transformation Network for Video Action Recognition

Effective spatiotemporal feature representation is crucial to the video-...
research
02/11/2018

Dual Control Memory Augmented Neural Networks for Treatment Recommendations

Machine-assisted treatment recommendations hold a promise to reduce phys...
research
05/12/2020

3DV: 3D Dynamic Voxel for Action Recognition in Depth Video

To facilitate depth-based 3D action recognition, 3D dynamic voxel (3DV) ...
research
02/17/2023

Video Action Recognition Collaborative Learning with Dynamics via PSO-ConvNet Transformer

Human Action Recognition (HAR) involves the task of categorizing actions...
research
02/28/2017

Scene Flow to Action Map: A New Representation for RGB-D based Action Recognition with Convolutional Neural Networks

Scene flow describes the motion of 3D objects in real world and potentia...
research
06/15/2020

Learn to cycle: Time-consistent feature discovery for action recognition

Temporal motion has been one of the essential components for effectively...

Please sign up or login with your details

Forgot password? Click here to reset