DeepAI
Log In Sign Up

STEP: Segmenting and Tracking Every Pixel

02/23/2021
by   Mark Weber, et al.
0

In this paper, we tackle video panoptic segmentation, a task that requires assigning semantic classes and track identities to all pixels in a video. To study this important problem in a setting that requires a continuous interpretation of sensory data, we present a new benchmark: Segmenting and Tracking Every Pixel (STEP), encompassing two datasets, KITTI-STEP, and MOTChallenge-STEP together with a new evaluation metric. Our work is the first that targets this task in a real-world setting that requires dense interpretation in both spatial and temporal domains. As the ground-truth for this task is difficult and expensive to obtain, existing datasets are either constructed synthetically or only sparsely annotated within short video clips. By contrast, our datasets contain long video sequences, providing challenging examples and a test-bed for studying long-term pixel-precise segmentation and tracking. For measuring the performance, we propose a novel evaluation metric Segmentation and Tracking Quality (STQ) that fairly balances semantic and tracking aspects of this task and is suitable for evaluating sequences of arbitrary length. We will make our datasets, metric, and baselines publicly available.

READ FULL TEXT

page 1

page 3

page 13

06/19/2020

Video Panoptic Segmentation

Panoptic segmentation has become a new standard of visual recognition ta...
12/09/2020

ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation

In this paper, we present ViP-DeepLab, a unified model attempting to tac...
09/26/2022

EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations

We introduce VISOR, a new dataset of pixel annotations and a benchmark s...
04/17/2020

MOPT: Multi-Object Panoptic Tracking

Comprehensive understanding of dynamic scenes is a critical prerequisite...
06/22/2021

VoxelEmbed: 3D Instance Segmentation and Tracking with Voxel Embedding based Deep Learning

Recent advances in bioimaging have provided scientists a superior high s...
08/07/2019

Visual Coin-Tracking: Tracking of Planar Double-Sided Objects

We introduce a new video analysis problem -- tracking of rigid planar ob...
12/13/2021

SVIP: Sequence VerIfication for Procedures in Videos

In this paper, we propose a novel sequence verification task that aims t...