Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation

by   Donghyeon Baek, et al.

We address the problem of generalized zero-shot semantic segmentation (GZS3) predicting pixel-wise semantic labels for seen and unseen classes. Most GZS3 methods adopt a generative approach that synthesizes visual features of unseen classes from corresponding semantic ones (e.g., word2vec) to train novel classifiers for both seen and unseen classes. Although generative methods show decent performance, they have two limitations: (1) the visual features are biased towards seen classes; (2) the classifier should be retrained whenever novel unseen classes appear. We propose a discriminative approach to address these limitations in a unified framework. To this end, we leverage visual and semantic encoders to learn a joint embedding space, where the semantic encoder transforms semantic features to semantic prototypes that act as centers for visual features of corresponding classes. Specifically, we introduce boundary-aware regression (BAR) and semantic consistency (SC) losses to learn discriminative features. Our approach to exploiting the joint embedding space, together with BAR and SC terms, alleviates the seen bias problem. At test time, we avoid the retraining process by exploiting semantic prototypes as a nearest-neighbor (NN) classifier. To further alleviate the bias problem, we also propose an inference technique, dubbed Apollonius calibration (AC), that modulates the decision boundary of the NN classifier to the Apollonius circle adaptively. Experimental results demonstrate the effectiveness of our framework, achieving a new state of the art on standard benchmarks.


page 4

page 6

page 14

page 15

page 16

page 17


Bias-Awareness for Zero-Shot Learning the Seen and Unseen

Generalized zero-shot learning recognizes inputs from both seen and unse...

Generative Dual Adversarial Network for Generalized Zero-shot Learning

This paper studies the problem of generalized zero-shot learning which r...

Entropy-Based Uncertainty Calibration for Generalized Zero-Shot Learning

Compared to conventional zero-shot learning (ZSL) where recognising unse...

Generalised Zero-Shot Learning with a Classifier Ensemble over Multi-Modal Embedding Spaces

Generalised zero-shot learning (GZSL) methods aim to classify previously...

Learning Semantic Ambiguities for Zero-Shot Learning

Zero-shot learning (ZSL) aims at recognizing classes for which no visual...

Attribute-Induced Bias Eliminating for Transductive Zero-Shot Learning

Transductive Zero-shot learning (ZSL) targets to recognize the unseen ca...

Segmenting 3D Hybrid Scenes via Zero-Shot Learning

This work is to tackle the problem of point cloud semantic segmentation ...

Please sign up or login with your details

Forgot password? Click here to reset