Semantically-Guided Representation Learning for Self-Supervised Monocular Depth

02/27/2020
by   Vitor Guizilini, et al.
5

Self-supervised learning is showing great promise for monocular depth estimation, using geometry as the only source of supervision. Depth networks are indeed capable of learning representations that relate visual appearance to 3D properties by implicitly leveraging category-level patterns. In this work we investigate how to leverage more directly this semantic structure to guide geometric representation learning, while remaining in the self-supervised regime. Instead of using semantic labels and proxy losses in a multi-task approach, we propose a new architecture leveraging fixed pretrained semantic segmentation networks to guide self-supervised representation learning via pixel-adaptive convolutions. Furthermore, we propose a two-stage training process to overcome a common semantic bias on dynamic objects via resampling. Our method improves upon the state of the art for self-supervised monocular depth prediction over all pixels, fine-grained details, and per semantic categories.

READ FULL TEXT

page 2

page 5

page 8

page 13

research
03/19/2021

Bootstrapped Self-Supervised Training with Monocular Video for Semantic Segmentation and Depth Estimation

For a robot deployed in the world, it is desirable to have the ability o...
research
10/14/2022

MonoDVPS: A Self-Supervised Monocular Depth Estimation Approach to Depth-aware Video Panoptic Segmentation

Depth-aware video panoptic segmentation tackles the inverse projection p...
research
10/24/2021

X-Distill: Improving Self-Supervised Monocular Depth via Cross-Task Distillation

In this paper, we propose a novel method, X-Distill, to improve the self...
research
09/30/2020

S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistency

A robot's ability to act is fundamentally constrained by what it can per...
research
01/07/2019

NVS Machines: Learning Novel View Synthesis with Fine-grained View Control

We present an approach that learns to synthesize high-quality, novel vie...
research
08/10/2020

SynDistNet: Self-Supervised Monocular Fisheye Camera Distance Estimation Synergized with Semantic Segmentation for Autonomous Driving

State-of-the-art self-supervised learning approaches for monocular depth...
research
04/14/2023

The Second Monocular Depth Estimation Challenge

This paper discusses the results for the second edition of the Monocular...

Please sign up or login with your details

Forgot password? Click here to reset