SPSG: Self-Supervised Photometric Scene Generation from RGB-D Scans

06/25/2020
by   Angela Dai, et al.
10

We present SPSG, a novel approach to generate high-quality, colored 3D models of scenes from RGB-D scan observations by learning to infer unobserved scene geometry and color in a self-supervised fashion. Our self-supervised approach learns to jointly inpaint geometry and color by correlating an incomplete RGB-D scan with a more complete version of that scan. Notably, rather than relying on 3D reconstruction losses to inform our 3D geometry and color reconstruction, we propose adversarial and perceptual losses operating on 2D renderings in order to achieve high-resolution, high-quality colored reconstructions of scenes. This exploits the high-resolution, self-consistent signal from individual raw RGB-D frames, in contrast to fused 3D reconstructions of the frames which exhibit inconsistencies from view-dependent effects, such as color balancing or pose inconsistencies. Thus, by informing our 3D scene generation directly through 2D signal, we produce high-quality colored reconstructions of 3D scenes, outperforming state of the art on both synthetic and real data.

READ FULL TEXT

page 2

page 5

page 7

page 8

page 12

page 13

page 14

research
11/29/2019

SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans

We present a novel approach that converts partial and noisy RGB-D scans ...
research
08/22/2023

ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes

We present ScanNet++, a large-scale dataset that couples together captur...
research
04/26/2019

3D-SIC: 3D Semantic Instance Completion for RGB-D Scans

This paper focuses on the task of semantic instance completion: from an ...
research
03/27/2016

3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions

Matching local geometric features on real-world depth images is a challe...
research
03/18/2020

Adversarial Texture Optimization from RGB-D Scans

Realistic color texture generation is an important step in RGB-D surface...
research
07/07/2021

Self-supervised Outdoor Scene Relighting

Outdoor scene relighting is a challenging problem that requires good und...
research
10/10/2022

Floorplan-Aware Camera Poses Refinement

Processing large indoor scenes is a challenging task, as scan registrati...

Please sign up or login with your details

Forgot password? Click here to reset