Log In Sign Up

Three Ways to Improve Semantic Segmentation with Self-Supervised Depth Estimation

by   Lukas Hoyer, et al.

Training deep networks for semantic segmentation requires large amounts of labeled training data, which presents a major challenge in practice, as labeling segmentation masks is a highly labor-intensive process. To address this issue, we present a framework for semi-supervised semantic segmentation, which is enhanced by self-supervised monocular depth estimation from unlabeled images. In particular, we propose three key contributions: (1) We transfer knowledge from features learned during self-supervised depth estimation to semantic segmentation, (2) we implement a strong data augmentation by blending images and labels using the structure of the scene, and (3) we utilize the depth feature diversity as well as the level of difficulty of learning depth in a student-teacher framework to select the most useful samples to be annotated for semantic segmentation. We validate the proposed model on the Cityscapes dataset, where all three modules demonstrate significant performance gains, and we achieve state-of-the-art results for semi-supervised semantic segmentation. The implementation is available at


page 4

page 5

page 6

page 8

page 13

page 14

page 15


Bootstrapped Self-Supervised Training with Monocular Video for Semantic Segmentation and Depth Estimation

For a robot deployed in the world, it is desirable to have the ability o...

Semantics-Depth-Symbiosis: Deeply Coupled Semi-Supervised Learning of Semantics and Depth

Multi-task learning (MTL) paradigm focuses on jointly learning two or mo...

False Negative Reduction in Semantic Segmentation under Domain Shift using Depth Estimation

State-of-the-art deep neural networks demonstrate outstanding performanc...

A comparative study of semi- and self-supervised semantic segmentation of biomedical microscopy data

In recent years, Convolutional Neural Networks (CNNs) have become the st...

OffRoadTranSeg: Semi-Supervised Segmentation using Transformers on OffRoad environments

We present OffRoadTranSeg, the first end-to-end framework for semi-super...

Self-Supervised Domain Mismatch Estimation for Autonomous Perception

Autonomous driving requires self awareness of its perception functions. ...

Joint Learning of Frequency and Spatial Domains for Dense Predictions

Current artificial neural networks mainly conduct the learning process i...

Code Repositories


An implementation of our work "Three Ways to Improve Semantic Segmentation with Self-Supervised Depth Estimation"

view repo