StereoGAN: Bridging Synthetic-to-Real Domain Gap by Joint Optimization of Domain Translation and Stereo Matching

05/05/2020
by   Rui Liu, et al.
7

Large-scale synthetic datasets are beneficial to stereo matching but usually introduce known domain bias. Although unsupervised image-to-image translation networks represented by CycleGAN show great potential in dealing with domain gap, it is non-trivial to generalize this method to stereo matching due to the problem of pixel distortion and stereo mismatch after translation. In this paper, we propose an end-to-end training framework with domain translation and stereo matching networks to tackle this challenge. First, joint optimization between domain translation and stereo matching networks in our end-to-end framework makes the former facilitate the latter one to the maximum extent. Second, this framework introduces two novel losses, i.e., bidirectional multi-scale feature re-projection loss and correlation consistency loss, to help translate all synthetic stereo images into realistic ones as well as maintain epipolar constraints. The effective combination of above two contributions leads to impressive stereo-consistent translation and disparity estimation accuracy. In addition, a mode seeking regularization term is added to endow the synthetic-to-real translation results with higher fine-grained diversity. Extensive experiments demonstrate the effectiveness of the proposed framework on bridging the synthetic-to-real domain gap on stereo matching.

READ FULL TEXT

page 1

page 2

research
09/04/2023

StereoFlowGAN: Co-training for Stereo and Flow with Unsupervised Domain Adaptation

We introduce a novel training strategy for stereo matching and optical f...
research
05/10/2020

Domain Adaptation for Image Dehazing

Image dehazing using learning-based methods has achieved state-of-the-ar...
research
03/26/2021

Geometry-Aware Unsupervised Domain Adaptation for Stereo Matching

Recently proposed DNN-based stereo matching methods that learn priors di...
research
03/04/2019

Unsupervised Cross-spectral Stereo Matching by Learning to Synthesize

Unsupervised cross-spectral stereo matching aims at recovering disparity...
research
08/04/2018

T2Net: Synthetic-to-Realistic Translation for Solving Single-Image Depth Estimation Tasks

Current methods for single-image depth estimation use training datasets ...
research
04/16/2023

Translating Simulation Images to X-ray Images via Multi-Scale Semantic Matching

Endovascular intervention training is increasingly being conducted in vi...
research
10/14/2020

Matching-space Stereo Networks for Cross-domain Generalization

End-to-end deep networks represent the state of the art for stereo match...

Please sign up or login with your details

Forgot password? Click here to reset