Segmenter: Transformer for Semantic Segmentation

05/12/2021
by   Robin Strudel, et al.
40

Image segmentation is often ambiguous at the level of individual image patches and requires contextual information to reach label consensus. In this paper we introduce Segmenter, a transformer model for semantic segmentation. In contrast to convolution based approaches, our approach allows to model global context already at the first layer and throughout the network. We build on the recent Vision Transformer (ViT) and extend it to semantic segmentation. To do so, we rely on the output embeddings corresponding to image patches and obtain class labels from these embeddings with a point-wise linear decoder or a mask transformer decoder. We leverage models pre-trained for image classification and show that we can fine-tune them on moderate sized datasets available for semantic segmentation. The linear decoder allows to obtain excellent results already, but the performance can be further improved by a mask transformer generating class masks. We conduct an extensive ablation study to show the impact of the different parameters, in particular the performance is better for large models and small patch sizes. Segmenter attains excellent results for semantic segmentation. It outperforms the state of the art on the challenging ADE20K dataset and performs on-par on Pascal Context and Cityscapes.

READ FULL TEXT

page 1

page 3

page 5

page 8

page 13

page 15

page 16

page 17

research
08/03/2022

SSformer: A Lightweight Transformer for Semantic Segmentation

It is well believed that Transformer performs better in semantic segment...
research
03/26/2023

Contrastive Transformer: Contrastive Learning Scheme with Transformer innate Patches

This paper presents Contrastive Transformer, a contrastive learning sche...
research
03/26/2022

Semantic Segmentation by Early Region Proxy

Typical vision backbones manipulate structured features. As a compromise...
research
11/24/2016

Recalling Holistic Information for Semantic Segmentation

Semantic segmentation requires a detailed labeling of image pixels by ob...
research
08/25/2023

A Re-Parameterized Vision Transformer (ReVT) for Domain-Generalized Semantic Segmentation

The task of semantic segmentation requires a model to assign semantic la...
research
01/28/2023

ProtoSeg: Interpretable Semantic Segmentation with Prototypical Parts

We introduce ProtoSeg, a novel model for interpretable semantic image se...
research
08/24/2023

Boosting Semantic Segmentation from the Perspective of Explicit Class Embeddings

Semantic segmentation is a computer vision task that associates a label ...

Please sign up or login with your details

Forgot password? Click here to reset