DeepAI
Log In Sign Up

Rethinking Resolution in the Context of Efficient Video Recognition

09/26/2022
by   Chuofan Ma, et al.
23

In this paper, we empirically study how to make the most of low-resolution frames for efficient video recognition. Existing methods mainly focus on developing compact networks or alleviating temporal redundancy of video inputs to increase efficiency, whereas compressing frame resolution has rarely been considered a promising solution. A major concern is the poor recognition accuracy on low-resolution frames. We thus start by analyzing the underlying causes of performance degradation on low-resolution frames. Our key finding is that the major cause of degradation is not information loss in the down-sampling process, but rather the mismatch between network architecture and input scale. Motivated by the success of knowledge distillation (KD), we propose to bridge the gap between network and input size via cross-resolution KD (ResKD). Our work shows that ResKD is a simple but effective method to boost recognition accuracy on low-resolution frames. Without bells and whistles, ResKD considerably surpasses all competitive methods in terms of efficiency and accuracy on four large-scale benchmark datasets, i.e., ActivityNet, FCVID, Mini-Kinetics, Something-Something V2. In addition, we extensively demonstrate its effectiveness over state-of-the-art architectures, i.e., 3D-CNNs and Video Transformers, and scalability towards super low-resolution frames. The results suggest ResKD can serve as a general inference acceleration method for state-of-the-art video recognition. Our code will be available at https://github.com/CVMI-Lab/ResKD.

READ FULL TEXT
12/02/2019

More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregation

Current state-of-the-art models for video action recognition are mostly ...
07/14/2022

Dynamic Low-Resolution Distillation for Cost-Efficient End-to-End Text Spotting

End-to-end text spotting has attached great attention recently due to it...
07/23/2020

MuCAN: Multi-Correspondence Aggregation Network for Video Super-Resolution

Video super-resolution (VSR) aims to utilize multiple low-resolution fra...
02/15/2020

Video Face Super-Resolution with Motion-Adaptive Feedback Cell

Video super-resolution (VSR) methods have recently achieved a remarkable...
03/29/2021

Video Classification with FineCoarse Networks

A rich representation of the information in video data can be realized b...
04/09/2020

X3D: Expanding Architectures for Efficient Video Recognition

This paper presents X3D, a family of efficient video networks that progr...
12/28/2021

AdaFocus V2: End-to-End Training of Spatial Dynamic Networks for Video Recognition

Recent works have shown that the computational efficiency of video recog...