MOPT: Multi-Object Panoptic Tracking

by   Juana Valeria Hurtado, et al.

Comprehensive understanding of dynamic scenes is a critical prerequisite for intelligent robots to autonomously operate in our environment. Research in this domain which encompasses diverse perception problems has primarily been limited to addressing specific tasks individually and thus has contributed very little towards modeling the ability to understand dynamic scenes holistically. As a step towards encouraging research in this direction, we introduce a new perception task that we name Multi-Object Panoptic Tracking (MOPT). MOPT unifies the conventionally disjoint tasks of semantic segmentation, instance segmentation, and multi-object tracking. MOPT allows for exploiting pixel-level semantic information of 'thing' and 'stuff' classes, temporal coherence, and pixel-level associations over time, for the mutual benefit of each of these sub-problems. In order to facilitate quantitative evaluations of MOPT in a unified manner, we propose the soft Panoptic Tracking Quality (sPTQ) metric. As a first step towards addressing this task, we propose the novel PanopticTrackNet architecture that builds upon the state-of-the-art top-down panoptic segmentation network EfficientPS by adding a new tracking head to simultaneously learn all subtasks in an end-to-end manner. Additionally, we present several strong baselines that combine predictions from state-of-the-art panoptic segmentation and multi-object tracking models for comparison. We present extensive quantitative and qualitative evaluations for both vision-based and LiDAR-based MOPT on the challenging Virtual KITTI 2 and SemanticKITTI datasets, which demonstrates encouraging results.


page 1

page 3

page 8


Amodal Panoptic Segmentation

Humans have the remarkable ability to perceive objects as a whole, even ...

Fast Video Object Segmentation With Temporal Aggregation Network and Dynamic Template Matching

Significant progress has been made in Video Object Segmentation (VOS), t...

4D Panoptic LiDAR Segmentation

Temporal semantic scene understanding is critical for self-driving cars ...

Weakly Supervised Multi-Object Tracking and Segmentation

We introduce the problem of weakly supervised Multi-Object Tracking and ...

STEP: Segmenting and Tracking Every Pixel

In this paper, we tackle video panoptic segmentation, a task that requir...

Panoptic nuScenes: A Large-Scale Benchmark for LiDAR Panoptic Segmentation and Tracking

Panoptic scene understanding and tracking of dynamic agents are essentia...

Tracking Accuracy Based Generation Rules of Collective Perception Messages

The Collective Perception Service (CPS) enables the enhancement of envir...

Code Repositories