Neural Rerendering in the Wild

by   Moustafa Meshry, et al.

We explore total scene capture -- recording, modeling, and rerendering a scene under varying appearance such as season and time of day. Starting from internet photos of a tourist landmark, we apply traditional 3D reconstruction to register the photos and approximate the scene as a point cloud. For each photo, we render the scene points into a deep framebuffer, and train a neural network to learn the mapping of these initial renderings to the actual photos. This rerendering network also takes as input a latent appearance vector and a semantic mask indicating the location of transient objects like pedestrians. The model is evaluated on several datasets of publicly available images spanning a broad range of illumination conditions. We create short videos demonstrating realistic manipulation of the image viewpoint, appearance, and semantic labeling. We also compare results with prior work on scene reconstruction from internet photos.



There are no comments yet.


page 5

page 6

page 7

page 8

page 9

page 11

page 12

page 13


Head Reconstruction from Internet Photos

3D face reconstruction from Internet photos has recently produced exciti...

CartoonRenderer: An Instance-based Multi-Style Cartoon Image Translator

Instance based photo cartoonization is one of the challenging image styl...

3D Time-lapse Reconstruction from Internet Photos

Given an Internet photo collection of a landmark, we compute a 3D time-l...

Manipulating Attributes of Natural Scenes via Hallucination

In this study, we explore building a two-stage framework for enabling us...

Plan2Scene: Converting Floorplans to 3D Scenes

We address the task of converting a floorplan and a set of associated ph...

Crowdsampling the Plenoptic Function

Many popular tourist landmarks are captured in a multitude of online, pu...

Selfie Drone Stick: A Natural Interface for Quadcopter Photography

A physical selfie stick extends the user's reach, enabling the creation ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.