DeepAI AI Chat
Log In Sign Up

Weakly supervised segmentation with cross-modality equivariant constraints

by   Gaurav patel, et al.

Weakly supervised learning has emerged as an appealing alternative to alleviate the need for large labeled datasets in semantic segmentation. Most current approaches exploit class activation maps (CAMs), which can be generated from image-level annotations. Nevertheless, resulting maps have been demonstrated to be highly discriminant, failing to serve as optimal proxy pixel-level labels. We present a novel learning strategy that leverages self-supervision in a multi-modal image scenario to significantly enhance original CAMs. In particular, the proposed method is based on two observations. First, the learning of fully-supervised segmentation networks implicitly imposes equivariance by means of data augmentation, whereas this implicit constraint disappears on CAMs generated with image tags. And second, the commonalities between image modalities can be employed as an efficient self-supervisory signal, correcting the inconsistency shown by CAMs obtained across multiple modalities. To effectively train our model, we integrate a novel loss function that includes a within-modality and a cross-modality equivariant term to explicitly impose these constraints during training. In addition, we add a KL-divergence on the class prediction distributions to facilitate the information exchange between modalities, which, combined with the equivariant regularizers further improves the performance of our model. Exhaustive experiments on the popular multi-modal BRATS dataset demonstrate that our approach outperforms relevant recent literature under the same learning conditions.


page 1

page 3

page 7

page 8


Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation

Image-level weakly supervised semantic segmentation is a challenging pro...

Harvesting Information from Captions for Weakly Supervised Semantic Segmentation

Since acquiring pixel-wise annotations for training convolutional neural...

Unpaired Multi-modal Segmentation via Knowledge Distillation

Multi-modal learning is typically performed with network architectures c...

Exploring Pixel-level Self-supervision for Weakly Supervised Semantic Segmentation

Existing studies in weakly supervised semantic segmentation (WSSS) have ...

Exploring Weakly Supervised Semantic Segmentation Ensembles for Medical Imaging Systems

Reliable classification and detection of certain medical conditions, in ...

CheXseg: Combining Expert Annotations with DNN-generated Saliency Maps for X-ray Segmentation

Medical image segmentation models are typically supervised by expert ann...

S2-Net: Self-supervision Guided Feature Representation Learning for Cross-Modality Images

Combining the respective advantages of cross-modality images can compens...