Dynamic Object Removal and Spatio-Temporal RGB-D Inpainting via Geometry-Aware Adversarial Learning

08/12/2020
by   Borna Bešić, et al.
0

Dynamic objects have a significant impact on the robot's perception of the environment which degrades the performance of essential tasks such as localization and mapping. In this work, we address this problem by synthesizing plausible color, texture and geometry in regions occluded by dynamic objects. We propose the novel geometry-aware DynaFill architecture that follows a coarse-to-fine topology and incorporates our gated recurrent feedback mechanism to adaptively fuse information from previous timesteps. We optimize our architecture using adversarial training to synthesize fine realistic textures which enables it to hallucinate color and depth structure in occluded regions online in a spatially and temporally coherent manner, without relying on future frame information. Casting our inpainting problem as an image-to-image translation task, our model also corrects regions correlated with the presence of dynamic objects in the scene, such as shadows or reflections. We introduce a large-scale hyperrealistic dataset with RGB-D images, semantic segmentation labels, camera poses as well as ground truth RGB-D information of occluded regions. Extensive quantitative and qualitative evaluations show that our approach achieves state-of-the-art performance, even in challenging weather conditions. Furthermore, we present results for retrieval-based visual localization with the synthesized images that demonstrate the utility of our approach.

READ FULL TEXT

page 1

page 6

page 7

page 10

page 12

page 13

page 14

page 15

research
10/14/2021

RGB-D Image Inpainting Using Generative Adversarial Network with a Late Fusion Approach

Diminished reality is a technology that aims to remove objects from vide...
research
10/15/2020

Empty Cities: a Dynamic-Object-Invariant Space for Visual SLAM

In this paper we present a data-driven approach to obtain the static ima...
research
04/10/2022

Scale Invariant Semantic Segmentation with RGB-D Fusion

In this paper, we propose a neural network architecture for scale-invari...
research
08/15/2021

Occlusion-Aware Video Object Inpainting

Conventional video inpainting is neither object-oriented nor occlusion-a...
research
09/20/2018

Empty Cities: Image Inpainting for a Dynamic-Object-Invariant Space

In this paper we present an end-to-end deep learning framework to turn i...
research
03/18/2020

Adversarial Texture Optimization from RGB-D Scans

Realistic color texture generation is an important step in RGB-D surface...
research
04/09/2020

3D Photography using Context-aware Layered Depth Inpainting

We propose a method for converting a single RGB-D input image into a 3D ...

Please sign up or login with your details

Forgot password? Click here to reset