Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation

07/25/2022
by   Jiaming Zhang, et al.
6

In this paper, we address panoramic semantic segmentation, which provides a full-view and dense-pixel understanding of surroundings in a holistic way. Panoramic segmentation is under-explored due to two critical challenges: (1) image distortions and object deformations on panoramas; (2) lack of annotations for training panoramic segmenters. To tackle these problems, we propose a Transformer for Panoramic Semantic Segmentation (Trans4PASS) architecture. First, to enhance distortion awareness, Trans4PASS, equipped with Deformable Patch Embedding (DPE) and Deformable MLP (DMLP) modules, is capable of handling object deformations and image distortions whenever (before or after adaptation) and wherever (shallow or deep levels) by design. We further introduce the upgraded Trans4PASS+ model, featuring DMLPv2 with parallel token mixing to improve the flexibility and generalizability in modeling discriminative cues. Second, we propose a Mutual Prototypical Adaptation (MPA) strategy for unsupervised domain adaptation. Third, aside from Pinhole-to-Panoramic (Pin2Pan) adaptation, we create a new dataset (SynPASS) with 9,080 panoramic images to explore a Synthetic-to-Real (Syn2Real) adaptation scheme in 360 imagery. Extensive experiments are conducted, which cover indoor and outdoor scenarios, and each of them is investigated with Pin2Pan and Syn2Real regimens. Trans4PASS+ achieves state-of-the-art performances on four domain adaptive panoramic semantic segmentation benchmarks. Code is available at https://github.com/jamycheung/Trans4PASS.

READ FULL TEXT

page 1

page 2

page 7

page 13

page 18

research
03/02/2022

Bending Reality: Distortion-aware Transformers for Adapting to Panoramic Semantic Segmentation

Panoramic images with their 360-degree directional view encompass exhaus...
research
06/06/2023

SGAT4PASS: Spherical Geometry-Aware Transformer for PAnoramic Semantic Segmentation

As an important and challenging problem in computer vision, PAnoramic Se...
research
03/07/2022

Semantic Segmentation in Art Paintings

Semantic segmentation is a difficult task even when trained in a supervi...
research
11/21/2022

Computational Optics Meet Domain Adaptation: Transferring Semantic Segmentation Beyond Aberrations

Semantic scene understanding with Minimalist Optical Systems (MOS) in mo...
research
04/27/2023

EDAPS: Enhanced Domain-Adaptive Panoptic Segmentation

With autonomous industries on the rise, domain adaptation of the visual ...
research
08/10/2023

Look at the Neighbor: Distortion-aware Unsupervised Domain Adaptation for Panoramic Semantic Segmentation

Endeavors have been recently made to transfer knowledge from the labeled...
research
11/20/2018

Sensor Adaptation for Improved Semantic Segmentation of Overhead Imagery

Semantic segmentation is a powerful method to facilitate visual scene un...

Please sign up or login with your details

Forgot password? Click here to reset