PanopticDepth: A Unified Framework for Depth-aware Panoptic Segmentation

06/01/2022
by   Naiyu Gao, et al.
0

This paper presents a unified framework for depth-aware panoptic segmentation (DPS), which aims to reconstruct 3D scene with instance-level semantics from one single image. Prior works address this problem by simply adding a dense depth regression head to panoptic segmentation (PS) networks, resulting in two independent task branches. This neglects the mutually-beneficial relations between these two tasks, thus failing to exploit handy instance-level semantic cues to boost depth accuracy while also producing sub-optimal depth maps. To overcome these limitations, we propose a unified framework for the DPS task by applying a dynamic convolution technique to both the PS and depth prediction tasks. Specifically, instead of predicting depth for all pixels at a time, we generate instance-specific kernels to predict depth and segmentation masks for each instance. Moreover, leveraging the instance-wise depth estimation scheme, we add additional instance-level depth cues to assist with supervising the depth learning via a new depth loss. Extensive experiments on Cityscapes-DPS and SemKITTI-DPS show the effectiveness and promise of our method. We hope our unified solution to DPS can lead a new paradigm in this area. Code is available at https://github.com/NaiyuGao/PanopticDepth.

READ FULL TEXT

page 1

page 4

page 7

page 8

research
12/09/2020

ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation

In this paper, we present ViP-DeepLab, a unified model attempting to tac...
research
07/25/2023

PlaneRecTR: Unified Query learning for 3D Plane Recovery from a Single View

3D plane recovery from a single image can usually be divided into severa...
research
01/05/2023

All in Tokens: Unifying Output Space of Visual Tasks via Soft Token

Unlike language tasks, where the output space is usually limited to a se...
research
05/30/2019

Learning Semantics-aware Distance Map with Semantics Layering Network for Amodal Instance Segmentation

In this work, we demonstrate yet another approach to tackle the amodal s...
research
07/17/2023

On Point Affiliation in Feature Upsampling

We introduce the notion of point affiliation into feature upsampling. By...
research
07/27/2020

A Unified Framework of Surrogate Loss by Refactoring and Interpolation

We introduce UniLoss, a unified framework to generate surrogate losses f...
research
12/10/2018

Attention-guided Unified Network for Panoptic Segmentation

This paper studies panoptic segmentation, a recently proposed task which...

Please sign up or login with your details

Forgot password? Click here to reset