Exploring Consistency in Cross-Domain Transformer for Domain Adaptive Semantic Segmentation

11/27/2022
by   Kaihong Wang, et al.
0

While transformers have greatly boosted performance in semantic segmentation, domain adaptive transformers are not yet well explored. We identify that the domain gap can cause discrepancies in self-attention. Due to this gap, the transformer attends to spurious regions or pixels, which deteriorates accuracy on the target domain. We propose to perform adaptation on attention maps with cross-domain attention layers that share features between the source and the target domains. Specifically, we impose consistency between predictions from cross-domain attention and self-attention modules to encourage similar distribution in the attention and output of the model across domains, i.e., attention-level and output-level alignment. We also enforce consistency in attention maps between different augmented views to further strengthen the attention-based alignment. Combining these two components, our method mitigates the discrepancy in attention maps across domains and further boosts the performance of the transformer under unsupervised domain adaptation settings. Our model outperforms the existing state-of-the-art baseline model on three widely used benchmarks, including GTAV-to-Cityscapes by 1.3 percent point (pp), Synthia-to-Cityscapes by 0.6 pp, and Cityscapes-to-ACDC by 1.1 pp, on average. Additionally, we verify the effectiveness and generalizability of our method through extensive experiments. Our code will be publicly available.

READ FULL TEXT

page 1

page 7

page 8

research
12/20/2022

ADAS: A Simple Active-and-Adaptive Baseline for Cross-Domain 3D Semantic Segmentation

State-of-the-art 3D semantic segmentation models are trained on the off-...
research
03/09/2020

Context-Aware Domain Adaptation in Semantic Segmentation

In this paper, we consider the problem of unsupervised domain adaptation...
research
02/27/2022

Attention-based Cross-Layer Domain Alignment for Unsupervised Domain Adaptation

Unsupervised domain adaptation (UDA) aims to learn transferable knowledg...
research
06/01/2022

Cross-domain Detection Transformer based on Spatial-aware and Semantic-aware Token Alignment

Detection transformers like DETR have recently shown promising performan...
research
05/23/2023

Source-Free Domain Adaptation for RGB-D Semantic Segmentation with Vision Transformers

With the increasing availability of depth sensors, multimodal frameworks...
research
08/21/2022

Depth-Assisted ResiDualGAN for Cross-Domain Aerial Images Semantic Segmentation

Unsupervised domain adaptation (UDA) is an approach to minimizing domain...
research
09/20/2023

GraphEcho: Graph-Driven Unsupervised Domain Adaptation for Echocardiogram Video Segmentation

Echocardiogram video segmentation plays an important role in cardiac dis...

Please sign up or login with your details

Forgot password? Click here to reset