Learning Semantic Segmentation from Synthetic Data: A Geometrically Guided Input-Output Adaptation Approach

by   Yuhua Chen, et al.

Recently, increasing attention has been drawn to training semantic segmentation models using synthetic data and computer-generated annotation. However, domain gap remains a major barrier and prevents models learned from synthetic data from generalizing well to real-world applications. In this work, we take the advantage of additional geometric information from synthetic data, a powerful yet largely neglected cue, to bridge the domain gap. Such geometric information can be generated easily from synthetic data, and is proven to be closely coupled with semantic information. With the geometric information, we propose a model to reduce domain shift on two levels: on the input level, we augment the traditional image translation network with the additional geometric information to translate synthetic images into realistic styles; on the output level, we build a task network which simultaneously performs depth estimation and semantic segmentation on the synthetic data. Meanwhile, we encourage the network to preserve correlation between depth and semantics by adversarial training on the output space. We then validate our method on two pairs of synthetic to real dataset: Virtual KITTI to KITTI, and SYNTHIA to Cityscapes, where we achieve a significant performance gain compared to the non-adapt baseline and methods using only semantic label. This demonstrates the usefulness of geometric information from synthetic data for cross-domain semantic segmentation.


page 1

page 3

page 8


Geometric Unsupervised Domain Adaptation for Semantic Segmentation

Simulators can efficiently generate large amounts of labeled synthetic d...

ProCST: Boosting Semantic Segmentation using Progressive Cyclic Style-Transfer

Using synthetic data for training neural networks that achieve good perf...

Synthetic Convolutional Features for Improved Semantic Segmentation

Recently, learning-based image synthesis has enabled to generate high-re...

Exploiting Semantics in Adversarial Training for Image-Level Domain Adaptation

Performance achievable by modern deep learning approaches are directly r...

Learning color space adaptation from synthetic to real images of cirrus clouds

Training on synthetic data is becoming popular in vision due to the conv...

Validation of Simulation-Based Testing: Bypassing Domain Shift with Label-to-Image Synthesis

Many machine learning applications can benefit from simulated data for s...

MetaPix: Domain Transfer for Semantic Segmentation by Meta Pixel Weighting

Training a deep neural model for semantic segmentation requires collecti...

Please sign up or login with your details

Forgot password? Click here to reset