Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding

11/04/2020
by   Mike Roberts, et al.
0

For many fundamental scene understanding tasks, it is difficult or impossible to obtain per-pixel ground truth labels from real images. We address this challenge by introducing Hypersim, a photorealistic synthetic dataset for holistic indoor scene understanding. To create our dataset, we leverage a large repository of synthetic scenes created by professional artists, and we generate 77,400 images of 461 indoor scenes with detailed per-pixel labels and corresponding ground truth geometry. Our dataset: (1) relies exclusively on publicly available 3D assets; (2) includes complete scene geometry, material information, and lighting information for every scene; (3) includes dense per-pixel semantic instance segmentations for every image; and (4) factors every image into diffuse reflectance, diffuse illumination, and a non-diffuse residual term that captures view-dependent lighting effects. Together, these features make our dataset well-suited for geometric learning problems that require direct 3D supervision, multi-task learning problems that require reasoning jointly over multiple input and output modalities, and inverse rendering problems. We analyze our dataset at the level of scenes, objects, and pixels, and we analyze costs in terms of money, annotation effort, and computation time. Remarkably, we find that it is possible to generate our entire dataset from scratch, for roughly half the cost of training a state-of-the-art natural language processing model. All the code we used to generate our dataset is available online.

READ FULL TEXT

page 1

page 4

page 5

page 6

page 7

research
07/25/2020

OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasets

Large-scale photorealistic datasets of indoor scenes, with ground truth ...
research
08/16/2022

Casual Indoor HDR Radiance Capture from Omnidirectional Images

We present PanoHDR-NeRF, a novel pipeline to casually capture a plausibl...
research
11/15/2022

Deep scene-scale material estimation from multi-view indoor captures

The movie and video game industries have adopted photogrammetry as a way...
research
01/08/2019

Neural Inverse Rendering of an Indoor Scene from a Single Image

Inverse rendering aims to estimate physical scene attributes (e.g., refl...
research
12/24/2019

Rendering Synthetic Objects into Legacy Photographs

We propose a method to realistically insert synthetic objects into exist...
research
03/23/2022

Self-supervised HDR Imaging from Motion and Exposure Cues

Recent High Dynamic Range (HDR) techniques extend the capabilities of cu...
research
10/28/2021

Towards Large-Scale Rendering of Simulated Crops for Synthetic Ground Truth Generation on Modular Supercomputers

Computer Vision problems deal with the semantic extraction of informatio...

Please sign up or login with your details

Forgot password? Click here to reset