MGSampler: An Explainable Sampling Strategy for Video Action Recognition

by   Yuan Zhi, et al.

Frame sampling is a fundamental problem in video action recognition due to the essential redundancy in time and limited computation resources. The existing sampling strategy often employs a fixed frame selection and lacks the flexibility to deal with complex variations in videos. In this paper, we present an explainable, adaptive, and effective frame sampler, called Motion-guided Sampler (MGSampler). Our basic motivation is that motion is an important and universal signal that can drive us to select frames from videos adaptively. Accordingly, we propose two important properties in our MGSampler design: motion sensitive and motion uniform. First, we present two different motion representations to enable us to efficiently distinguish the motion salient frames from the background. Then, we devise a motion-uniform sampling strategy based on the cumulative motion distribution to ensure the sampled frames evenly cover all the important frames with high motion saliency. Our MGSampler yields a new principled and holistic sample scheme, that could be incorporated into any existing video architecture. Experiments on five benchmarks demonstrate the effectiveness of our MGSampler over previously fixed sampling strategies, and also its generalization power across different backbones, video models, and datasets.



page 2

page 3

page 5


Random Temporal Skipping for Multirate Video Analysis

Current state-of-the-art approaches to video understanding adopt tempora...

OCSampler: Compressing Videos to One Clip with Single-step Sampling

In this paper, we propose a framework named OCSampler to explore a compa...

SMART Frame Selection for Action Recognition

Action recognition is computationally expensive. In this paper, we addre...

TSI: Temporal Saliency Integration for Video Action Recognition

Efficient spatiotemporal modeling is an important yet challenging proble...

Challenge report:VIPriors Action Recognition Challenge

This paper is a brief report to our submission to the VIPriors Action Re...

Adversarially Robust Frame Sampling with Bounded Irregularities

In recent years, video analysis tools for automatically extracting meani...

EgoSampling: Fast-Forward and Stereo for Egocentric Videos

While egocentric cameras like GoPro are gaining popularity, the videos t...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.