Log In Sign Up

STEP: Segmenting and Tracking Every Pixel

by   Mark Weber, et al.

In this paper, we tackle video panoptic segmentation, a task that requires assigning semantic classes and track identities to all pixels in a video. To study this important problem in a setting that requires a continuous interpretation of sensory data, we present a new benchmark: Segmenting and Tracking Every Pixel (STEP), encompassing two datasets, KITTI-STEP, and MOTChallenge-STEP together with a new evaluation metric. Our work is the first that targets this task in a real-world setting that requires dense interpretation in both spatial and temporal domains. As the ground-truth for this task is difficult and expensive to obtain, existing datasets are either constructed synthetically or only sparsely annotated within short video clips. By contrast, our datasets contain long video sequences, providing challenging examples and a test-bed for studying long-term pixel-precise segmentation and tracking. For measuring the performance, we propose a novel evaluation metric Segmentation and Tracking Quality (STQ) that fairly balances semantic and tracking aspects of this task and is suitable for evaluating sequences of arbitrary length. We will make our datasets, metric, and baselines publicly available.


page 1

page 3

page 13


Video Panoptic Segmentation

Panoptic segmentation has become a new standard of visual recognition ta...

ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation

In this paper, we present ViP-DeepLab, a unified model attempting to tac...

EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations

We introduce VISOR, a new dataset of pixel annotations and a benchmark s...

MOPT: Multi-Object Panoptic Tracking

Comprehensive understanding of dynamic scenes is a critical prerequisite...

VoxelEmbed: 3D Instance Segmentation and Tracking with Voxel Embedding based Deep Learning

Recent advances in bioimaging have provided scientists a superior high s...

Visual Coin-Tracking: Tracking of Planar Double-Sided Objects

We introduce a new video analysis problem -- tracking of rigid planar ob...

SVIP: Sequence VerIfication for Procedures in Videos

In this paper, we propose a novel sequence verification task that aims t...