Clockwork Convnets for Video Semantic Segmentation

08/11/2016
by   Evan Shelhamer, et al.
0

Recent years have seen tremendous progress in still-image segmentation; however the naïve application of these state-of-the-art algorithms to every video frame requires considerable computation and ignores the temporal continuity inherent in video. We propose a video recognition framework that relies on two key observations: 1) while pixels may change rapidly from frame to frame, the semantic content of a scene evolves more slowly, and 2) execution can be viewed as an aspect of architecture, yielding purpose-fit computation schedules for networks. We define a novel family of "clockwork" convnets driven by fixed or adaptive clock signals that schedule the processing of different layers at different update rates according to their semantic stability. We design a pipeline schedule to reduce latency for real-time recognition and a fixed-rate schedule to reduce overall computation. Finally, we extend clockwork scheduling to adaptive video processing by incorporating data-driven clocks that can be tuned on unlabeled video. The accuracy and efficiency of clockwork convnets are evaluated on the Youtube-Objects, NYUD, and Cityscapes video datasets.

READ FULL TEXT

Authors

page 2

page 5

page 12

page 14

04/02/2018

Low-Latency Video Semantic Segmentation

Recent years have seen remarkable progress in semantic segmentation. Yet...
12/26/2019

Efficient Video Semantic Segmentation with Labels Propagation and Refinement

This paper tackles the problem of real-time semantic segmentation of hig...
08/29/2019

Exploiting Temporality for Semi-Supervised Video Segmentation

In recent years, there has been remarkable progress in supervised image ...
10/29/2019

Sequential image processing methods for improving semantic video segmentation algorithms

Recently, semantic video segmentation gained high attention especially f...
06/11/2018

Massively Parallel Video Networks

We introduce a class of causal video understanding models that aims to i...
04/06/2020

Fair Latency-Aware Metric for real-time video segmentation networks

As supervised semantic segmentation is reaching satisfying results, many...
07/15/2021

Real-Time Violence Detection Using CNN-LSTM

Violence rates however have been brought down about 57 the past 4 decade...

Code Repositories

clockwork-fcn

Clockwork Convnets for Video Semantic Segmenation


view repo

clockwork-kitti

Clockwork FCN in kitti dataset


view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.