Language-driven Semantic Segmentation

01/10/2022
by   Boyi Li, et al.
7

We present LSeg, a novel model for language-driven semantic image segmentation. LSeg uses a text encoder to compute embeddings of descriptive input labels (e.g., "grass" or "building") together with a transformer-based image encoder that computes dense per-pixel embeddings of the input image. The image encoder is trained with a contrastive objective to align pixel embeddings to the text embedding of the corresponding semantic class. The text embeddings provide a flexible label representation in which semantically similar labels map to similar regions in the embedding space (e.g., "cat" and "furry"). This allows LSeg to generalize to previously unseen categories at test time, without retraining or even requiring a single additional training sample. We demonstrate that our approach achieves highly competitive zero-shot performance compared to existing zero- and few-shot semantic segmentation methods, and even matches the accuracy of traditional segmentation algorithms when a fixed label set is provided. Code and demo are available at https://github.com/isl-org/lang-seg.

READ FULL TEXT

page 2

page 7

page 9

research
01/28/2023

ZegOT: Zero-shot Segmentation Through Optimal Transport of Text Prompts

Recent success of large-scale Contrastive Language-Image Pre-training (C...
research
12/18/2021

Prompt-Based Multi-Modal Image Segmentation

Image segmentation is usually addressed by training a model for a fixed ...
research
12/01/2022

Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairs

We tackle open-world semantic segmentation, which aims at learning to se...
research
12/20/2022

TeSS: Zero-Shot Classification via Textual Similarity Comparison with Prompting using Sentence Encoder

We introduce TeSS (Text Similarity Comparison using Sentence Encoder), a...
research
04/03/2023

Zero-Shot Semantic Segmentation with Decoupled One-Pass Network

Recently, the zero-shot semantic segmentation problem has attracted incr...
research
03/17/2022

Label conditioned segmentation

Semantic segmentation is an important task in computer vision that is of...
research
03/07/2021

Repurposing GANs for One-shot Semantic Part Segmentation

While GANs have shown success in realistic image generation, the idea of...

Please sign up or login with your details

Forgot password? Click here to reset