Live Intrinsic Material Estimation

01/03/2018
by   Abhimitra Meka, et al.
0

We present the first end-to-end approach for real-time material estimation for general object shapes that only requires a single color image as input. In addition to Lambertian surface properties, our approach fully automatically computes the specular albedo, material shininess, and a foreground segmentation. We tackle this challenging and ill-posed inverse rendering problem using recent advances in image-to-image translation techniques based on deep convolutional encoder-decoder architectures. The underlying core representations of our approach are specular shading, diffuse shading and mirror images, which allow to learn the effective and accurate separation of diffuse and specular albedo. In addition, we propose a novel highly efficient perceptual rendering loss that mimics real-world image formation and obtains intermediate results even during run time. The estimation of material parameters at real-time frame rates enables exciting mixed-reality applications, such as seamless illumination-consistent integration of virtual objects into real-world scenes, and virtual material cloning. We demonstrate our approach in a live setup, compare it to the state of the art, and motivate its effectiveness through quantitative and qualitative evaluation.

READ FULL TEXT

page 1

page 6

page 8

page 11

page 14

page 15

page 16

page 17

research
09/14/2023

OpenIllumination: A Multi-Illumination Dataset for Inverse Rendering Evaluation on Real Objects

We introduce OpenIllumination, a real-world dataset containing over 108K...
research
06/07/2019

Structural Decompositions for End-to-End Relighting

Relighting is an essential step in artificially transferring an object f...
research
11/21/2020

HDR Environment Map Estimation for Real-Time Augmented Reality

We present a method to estimate an HDR environment map from a narrow fie...
research
08/01/2017

Material Editing Using a Physically Based Rendering Network

The ability to edit materials of objects in images is desirable by many ...
research
07/24/2020

The Surprising Effectiveness of Linear Unsupervised Image-to-Image Translation

Unsupervised image-to-image translation is an inherently ill-posed probl...
research
12/05/2021

Neural Photometry-guided Visual Attribute Transfer

We present a deep learning-based method for propagating spatially-varyin...
research
03/14/2017

A fully end-to-end deep learning approach for real-time simultaneous 3D reconstruction and material recognition

This paper addresses the problem of simultaneous 3D reconstruction and m...

Please sign up or login with your details

Forgot password? Click here to reset