Dynamic Scene Novel View Synthesis via Deferred Spatio-temporal Consistency

by   Beatrix-Emőke Fülöp-Balogh, et al.

Structure from motion (SfM) enables us to reconstruct a scene via casual capture from cameras at different viewpoints, and novel view synthesis (NVS) allows us to render a captured scene from a new viewpoint. Both are hard with casual capture and dynamic scenes: SfM produces noisy and spatio-temporally sparse reconstructed point clouds, resulting in NVS with spatio-temporally inconsistent effects. We consider SfM and NVS parts together to ease the challenge. First, for SfM, we recover stable camera poses, then we defer the requirement for temporally-consistent points across the scene and reconstruct only a sparse point cloud per timestep that is noisy in space-time. Second, for NVS, we present a variational diffusion formulation on depths and colors that lets us robustly cope with the noise by enforcing spatio-temporal consistency via per-pixel reprojection weights derived from the input views. Together, this deferred approach generates novel views for dynamic scenes without requiring challenging spatio-temporally consistent reconstructions nor training complex models on large datasets. We demonstrate our algorithm on real-world dynamic scenes against classic and more recent learning-based baseline approaches.



There are no comments yet.


page 5

page 6

page 8

page 9

page 10

page 11

page 12


Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes

We present a method to perform novel view and time synthesis of dynamic ...

NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images

Neural Radiance Fields (NeRF) is a technique for high quality novel view...

Social-Sensor Composition for Tapestry Scenes

The extensive use of social media platforms and overwhelming amounts of ...

Learning to Factorize and Relight a City

We propose a learning-based framework for disentangling outdoor scenes i...

Purely Geometric Scene Association and Retrieval - A Case for Macro Scale 3D Geometry

We address the problems of measuring geometric similarity between 3D sce...

Learning Dynamic View Synthesis With Few RGBD Cameras

There have been significant advancements in dynamic novel view synthesis...

A Fast Free-viewpoint Video Synthesis Algorithm for Sports Scenes

In this paper, we report on a parallel freeviewpoint video synthesis alg...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.