DeepAI AI Chat
Log In Sign Up

Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding

by   Mike Roberts, et al.

For many fundamental scene understanding tasks, it is difficult or impossible to obtain per-pixel ground truth labels from real images. We address this challenge by introducing Hypersim, a photorealistic synthetic dataset for holistic indoor scene understanding. To create our dataset, we leverage a large repository of synthetic scenes created by professional artists, and we generate 77,400 images of 461 indoor scenes with detailed per-pixel labels and corresponding ground truth geometry. Our dataset: (1) relies exclusively on publicly available 3D assets; (2) includes complete scene geometry, material information, and lighting information for every scene; (3) includes dense per-pixel semantic instance segmentations for every image; and (4) factors every image into diffuse reflectance, diffuse illumination, and a non-diffuse residual term that captures view-dependent lighting effects. Together, these features make our dataset well-suited for geometric learning problems that require direct 3D supervision, multi-task learning problems that require reasoning jointly over multiple input and output modalities, and inverse rendering problems. We analyze our dataset at the level of scenes, objects, and pixels, and we analyze costs in terms of money, annotation effort, and computation time. Remarkably, we find that it is possible to generate our entire dataset from scratch, for roughly half the cost of training a state-of-the-art natural language processing model. All the code we used to generate our dataset is available online.


page 1

page 4

page 5

page 6

page 7


OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasets

Large-scale photorealistic datasets of indoor scenes, with ground truth ...

Casual Indoor HDR Radiance Capture from Omnidirectional Images

We present PanoHDR-NeRF, a novel pipeline to casually capture a plausibl...

Deep scene-scale material estimation from multi-view indoor captures

The movie and video game industries have adopted photogrammetry as a way...

Rendering Synthetic Objects into Legacy Photographs

We propose a method to realistically insert synthetic objects into exist...

Factorized Inverse Path Tracing for Efficient and Accurate Material-Lighting Estimation

Inverse path tracing has recently been applied to joint material and lig...

IRISformer: Dense Vision Transformers for Single-Image Inverse Rendering in Indoor Scenes

Indoor scenes exhibit significant appearance variations due to myriad in...