On the Integration of Optical Flow and Action Recognition

by   Laura Sevilla-Lara, et al.

Most of the top performing action recognition methods use optical flow as a "black box" input. Here we take a deeper look at the combination of flow and action recognition, and investigate why optical flow is helpful, what makes a flow method good for action recognition, and how we can make it better. In particular, we investigate the impact of different flow algorithms and input transformations to better understand how these affect a state-of-the-art action recognition method. Furthermore, we fine tune two neural-network flow methods end-to-end on the most widely used action recognition dataset (UCF101). Based on these experiments, we make the following five observations: 1) optical flow is useful for action recognition because it is invariant to appearance, 2) optical flow methods are optimized to minimize end-point-error (EPE), but the EPE of current methods is not well correlated with action recognition performance, 3) for the flow methods tested, accuracy at boundaries and at small displacements is most correlated with action recognition performance, 4) training optical flow to minimize classification error instead of minimizing EPE improves recognition performance, and 5) optical flow learned for the task of action recognition differs from traditional optical flow especially inside the human body and at the boundary of the body. These observations may encourage optical flow researchers to look beyond EPE as a goal and guide action recognition researchers to seek better motion cues, leading to a tighter integration of the optical flow and action recognition communities.


page 2

page 4

page 8


Combining Spatio-Temporal Appearance Descriptors and Optical Flow for Human Action Recognition in Video Data

This paper proposes combining spatio-temporal appearance (STA) descripto...

ActionFlowNet: Learning Motion Representation for Action Recognition

Even with the recent advances in convolutional neural networks (CNN) in ...

Representation Flow for Action Recognition

In this paper, we propose a convolutional layer inspired by optical flow...

Unsupervised Motion Representation Enhanced Network for Action Recognition

Learning reliable motion representation between consecutive frames, such...

Temporal Hockey Action Recognition via Pose and Optical Flows

Recognizing actions in ice hockey using computer vision poses challenges...

Action Recognition in the Frequency Domain

In this paper, we describe a simple strategy for mitigating variability ...

Prompt Learning for Action Recognition

We present a new general learning approach for action recognition, Promp...

Please sign up or login with your details

Forgot password? Click here to reset