Clockwork Convnets for Video Semantic Segmentation

by   Evan Shelhamer, et al.

Recent years have seen tremendous progress in still-image segmentation; however the naïve application of these state-of-the-art algorithms to every video frame requires considerable computation and ignores the temporal continuity inherent in video. We propose a video recognition framework that relies on two key observations: 1) while pixels may change rapidly from frame to frame, the semantic content of a scene evolves more slowly, and 2) execution can be viewed as an aspect of architecture, yielding purpose-fit computation schedules for networks. We define a novel family of "clockwork" convnets driven by fixed or adaptive clock signals that schedule the processing of different layers at different update rates according to their semantic stability. We design a pipeline schedule to reduce latency for real-time recognition and a fixed-rate schedule to reduce overall computation. Finally, we extend clockwork scheduling to adaptive video processing by incorporating data-driven clocks that can be tuned on unlabeled video. The accuracy and efficiency of clockwork convnets are evaluated on the Youtube-Objects, NYUD, and Cityscapes video datasets.



page 2

page 5

page 12

page 14


Low-Latency Video Semantic Segmentation

Recent years have seen remarkable progress in semantic segmentation. Yet...

Efficient Video Semantic Segmentation with Labels Propagation and Refinement

This paper tackles the problem of real-time semantic segmentation of hig...

Exploiting Temporality for Semi-Supervised Video Segmentation

In recent years, there has been remarkable progress in supervised image ...

Sequential image processing methods for improving semantic video segmentation algorithms

Recently, semantic video segmentation gained high attention especially f...

Massively Parallel Video Networks

We introduce a class of causal video understanding models that aims to i...

Fair Latency-Aware Metric for real-time video segmentation networks

As supervised semantic segmentation is reaching satisfying results, many...

Real-Time Violence Detection Using CNN-LSTM

Violence rates however have been brought down about 57 the past 4 decade...

Code Repositories


Clockwork Convnets for Video Semantic Segmenation

view repo


Clockwork FCN in kitti dataset

view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.