Calibrated Adversarial Refinement for Multimodal Semantic Segmentation

06/23/2020 ∙ by Elias Kassapis, et al. ∙ 0

Ambiguities in images or unsystematic annotation can lead to multiple valid solutions in semantic segmentation. To learn a distribution over predictions, recent work has explored the use of probabilistic networks. However, these do not necessarily capture the empirical distribution accurately. In this work, we aim to learn a calibrated multimodal predictive distribution, where the empirical frequency of the sampled predictions closely reflects that of the corresponding labels in the training set. To this end, we propose a novel two-stage, cascaded strategy for calibrated adversarial refinement. In the first stage, we explicitly model the data with a categorical likelihood. In the second, we train an adversarial network to sample from it an arbitrary number of coherent predictions. The model can be used independently or integrated into any black-box segmentation framework to enable the synthesis of diverse predictions. We demonstrate the utility and versatility of the approach by achieving competitive results on the multigrader LIDC dataset and a modified Cityscapes dataset. In addition, we use a toy regression dataset to show that our framework is not confined to semantic segmentation, and the core design can be adapted to other tasks requiring learning a calibrated predictive distribution.



There are no comments yet.


page 7

page 8

page 16

page 17

page 19

page 20

Code Repositories


Official re-implementation of the Calibrated Adversarial Refinement model described in the paper "Calibrated Adversarial Refinement for Multimodal Semantic Segmentation"

view repo


Reproducing the toy example in "Calibrated Adversarial Refinement for Multimodal Semantic Segmentation" by Kassapis et al.

view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.