Dynamic Scene Novel View Synthesis via Deferred Spatio-temporal Consistency

09/02/2021
by   Beatrix-Emőke Fülöp-Balogh, et al.
1

Structure from motion (SfM) enables us to reconstruct a scene via casual capture from cameras at different viewpoints, and novel view synthesis (NVS) allows us to render a captured scene from a new viewpoint. Both are hard with casual capture and dynamic scenes: SfM produces noisy and spatio-temporally sparse reconstructed point clouds, resulting in NVS with spatio-temporally inconsistent effects. We consider SfM and NVS parts together to ease the challenge. First, for SfM, we recover stable camera poses, then we defer the requirement for temporally-consistent points across the scene and reconstruct only a sparse point cloud per timestep that is noisy in space-time. Second, for NVS, we present a variational diffusion formulation on depths and colors that lets us robustly cope with the noise by enforcing spatio-temporal consistency via per-pixel reprojection weights derived from the input views. Together, this deferred approach generates novel views for dynamic scenes without requiring challenging spatio-temporally consistent reconstructions nor training complex models on large datasets. We demonstrate our algorithm on real-world dynamic scenes against classic and more recent learning-based baseline approaches.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 5

page 6

page 8

page 9

page 10

page 11

page 12

11/26/2020

Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes

We present a method to perform novel view and time synthesis of dynamic ...
11/26/2021

NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images

Neural Radiance Fields (NeRF) is a technique for high quality novel view...
03/28/2020

Social-Sensor Composition for Tapestry Scenes

The extensive use of social media platforms and overwhelming amounts of ...
08/06/2020

Learning to Factorize and Relight a City

We propose a learning-based framework for disentangling outdoor scenes i...
08/03/2018

Purely Geometric Scene Association and Retrieval - A Case for Macro Scale 3D Geometry

We address the problems of measuring geometric similarity between 3D sce...
04/22/2022

Learning Dynamic View Synthesis With Few RGBD Cameras

There have been significant advancements in dynamic novel view synthesis...
03/28/2019

A Fast Free-viewpoint Video Synthesis Algorithm for Sports Scenes

In this paper, we report on a parallel freeviewpoint video synthesis alg...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.