DeepAI AI Chat
Log In Sign Up

Fast Semantic Segmentation on Video Using Motion Vector-Based Feature Interpolation

by   Samvit Jain, et al.
berkeley college

Models optimized for accuracy on challenging, dense prediction tasks such as semantic segmentation entail significant inference costs, and are prohibitively slow to run on each frame in a video. Since nearby video frames are spatially similar, however, there is substantial opportunity to reuse computation. Existing work has explored basic feature reuse and feature warping based on optical flow, but has encountered limits to the speedup attainable with these techniques. In this paper, we present a new, two part approach to accelerating inference on video. Firstly, we propose a fast feature propagation scheme that utilizes the block motion vector maps present in compressed video to cheaply propagate features from frame to frame. Secondly, we develop a novel feature estimation scheme, termed feature interpolation, that fuses features propagated from enclosing keyframes to render accurate feature estimates, even at sparse keyframe frequencies. We evaluate our system on the Cityscapes dataset, comparing to both a frame-by-frame baseline and related work. We find that we are able to substantially accelerate segmentation on video, achieving almost twice the average inference speed as prior work at any target accuracy level.


page 2

page 13


Inter-BMV: Interpolation with Block Motion Vectors for Fast Semantic Segmentation on Video

Models optimized for accuracy on single images are often prohibitively s...

Video Semantic Segmentation with Distortion-Aware Feature Correction

Video semantic segmentation is active in recent years benefited from the...

BlockCopy: High-Resolution Video Processing with Block-Sparse Feature Propagation and Online Policies

In this paper we propose BlockCopy, a scheme that accelerates pretrained...

Accel: A Corrective Fusion Network for Efficient Semantic Segmentation on Video

In this paper, we present Accel, a novel semantic video segmentation sys...

How to Train Your Dragon: Tamed Warping Network for Semantic Video Segmentation

Real-time semantic segmentation on high-resolution videos is challenging...

Joint Forecasting of Features and Feature Motion for Dense Semantic Future Prediction

We present a novel dense semantic forecasting approach which is applicab...

Error-Aware Spatial Ensembles for Video Frame Interpolation

Video frame interpolation (VFI) algorithms have improved considerably in...