Transferring and Regularizing Prediction for Semantic Segmentation

06/11/2020
by   Yiheng Zhang, et al.
0

Semantic segmentation often requires a large set of images with pixel-level annotations. In the view of extremely expensive expert labeling, recent research has shown that the models trained on photo-realistic synthetic data (e.g., computer games) with computer-generated annotations can be adapted to real images. Despite this progress, without constraining the prediction on real images, the models will easily overfit on synthetic data due to severe domain mismatch. In this paper, we novelly exploit the intrinsic properties of semantic segmentation to alleviate such problem for model transfer. Specifically, we present a Regularizer of Prediction Transfer (RPT) that imposes the intrinsic properties as constraints to regularize model transfer in an unsupervised fashion. These constraints include patch-level, cluster-level and context-level semantic prediction consistencies at different levels of image formation. As the transfer is label-free and data-driven, the robustness of prediction is addressed by selectively involving a subset of image regions for model regularization. Extensive experiments are conducted to verify the proposal of RPT on the transfer of models trained on GTA5 and SYNTHIA (synthetic data) to Cityscapes dataset (urban street scenes). RPT shows consistent improvements when injecting the constraints on several neural networks for semantic segmentation. More remarkably, when integrating RPT into the adversarial-based segmentation framework, we report to-date the best results: mIoU of 53.2 respectively.

READ FULL TEXT

page 2

page 3

page 5

page 7

page 8

research
04/23/2018

Fully Convolutional Adaptation Networks for Semantic Segmentation

The recent advances in deep neural networks have convincingly demonstrat...
research
10/05/2021

MetaPix: Domain Transfer for Semantic Segmentation by Meta Pixel Weighting

Training a deep neural model for semantic segmentation requires collecti...
research
08/07/2016

Playing for Data: Ground Truth from Computer Games

Recent progress in computer vision has been driven by high-capacity mode...
research
07/16/2018

Effective Use of Synthetic Data for Urban Scene Semantic Segmentation

Training a deep network to perform semantic segmentation requires large ...
research
11/18/2017

Transferable Semi-supervised Semantic Segmentation

The performance of deep learning based semantic segmentation models heav...
research
05/12/2019

Style transfer based data augmentation in material microscopic image processing

Recently progress in material microscopic image semantic segmentation ha...
research
06/10/2021

Validation of Simulation-Based Testing: Bypassing Domain Shift with Label-to-Image Synthesis

Many machine learning applications can benefit from simulated data for s...

Please sign up or login with your details

Forgot password? Click here to reset