SMART Frame Selection for Action Recognition

12/19/2020
by   Shreyank N Gowda, et al.
0

Action recognition is computationally expensive. In this paper, we address the problem of frame selection to improve the accuracy of action recognition. In particular, we show that selecting good frames helps in action recognition performance even in the trimmed videos domain. Recent work has successfully leveraged frame selection for long, untrimmed videos, where much of the content is not relevant, and easy to discard. In this work, however, we focus on the more standard short, trimmed action recognition problem. We argue that good frame selection can not only reduce the computational cost of action recognition but also increase the accuracy by getting rid of frames that are hard to classify. In contrast to previous work, we propose a method that instead of selecting frames by considering one at a time, considers them jointly. This results in a more efficient selection, where good frames are more effectively distributed over the video, like snapshots that tell a story. We call the proposed frame selection SMART and we test it in combination with different backbone architectures and on multiple benchmarks (Kinetics, Something-something, UCF101). We show that the SMART frame selection consistently improves the accuracy compared to other frame selection strategies while reducing the computational cost by a factor of 4 to 10 times. Additionally, we show that when the primary goal is recognition performance, our selection strategy can improve over recent state-of-the-art models and frame selection strategies on various benchmarks (UCF101, HMDB51, FCVID, and ActivityNet).

READ FULL TEXT

page 3

page 6

research
04/20/2023

Search-Map-Search: A Frame Selection Paradigm for Action Recognition

Despite the success of deep learning in video understanding tasks, proce...
research
07/12/2021

Delta Sampling R-BERT for limited data and low-light action recognition

We present an approach to perform supervised action recognition in the d...
research
07/27/2023

Sample Less, Learn More: Efficient Action Recognition via Frame Feature Restoration

Training an effective video action recognition model poses significant c...
research
04/14/2023

PMI Sampler: Patch similarity guided frame selection for Aerial Action Recognition

We present a new algorithm for selection of informative frames in video ...
research
07/04/2020

Complex Human Action Recognition in Live Videos Using Hybrid FR-DL Method

Automated human action recognition is one of the most attractive and pra...
research
07/16/2020

Challenge report:VIPriors Action Recognition Challenge

This paper is a brief report to our submission to the VIPriors Action Re...
research
03/29/2021

No frame left behind: Full Video Action Recognition

Not all video frames are equally informative for recognizing an action. ...

Please sign up or login with your details

Forgot password? Click here to reset