DeepAI AI Chat
Log In Sign Up

Deformable Neural Radiance Fields

11/25/2020
by   Keunhong Park, et al.
22

We present the first method capable of photorealistically reconstructing a non-rigidly deforming scene using photos/videos captured casually from mobile phones. Our approach – D-NeRF – augments neural radiance fields (NeRF) by optimizing an additional continuous volumetric deformation field that warps each observed point into a canonical 5D NeRF. We observe that these NeRF-like deformation fields are prone to local minima, and propose a coarse-to-fine optimization method for coordinate-based models that allows for more robust optimization. By adapting principles from geometry processing and physical simulation to NeRF-like models, we propose an elastic regularization of the deformation field that further improves robustness. We show that D-NeRF can turn casually captured selfie photos/videos into deformable NeRF models that allow for photorealistic renderings of the subject from arbitrary viewpoints, which we dub "nerfies." We evaluate our method by collecting data using a rig with two mobile phones that take time-synchronized photos, yielding train/validation images of the same pose at different viewpoints. We show that our method faithfully reconstructs non-rigidly deforming scenes and reproduces unseen views with high fidelity.

READ FULL TEXT

page 1

page 4

page 5

page 6

page 7

page 9

page 10

07/19/2022

NDF: Neural Deformable Fields for Dynamic Human Modelling

We propose Neural Deformable Fields (NDF), a new representation for dyna...
06/24/2021

HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields

Neural Radiance Fields (NeRF) are able to reconstruct scenes with unprec...
05/31/2022

DeVRF: Fast Deformable Voxel Radiance Fields for Dynamic Scenes

Modeling dynamic scenes is important for many applications such as virtu...
03/24/2023

Deformable Model Driven Neural Rendering for High-fidelity 3D Reconstruction of Human Heads Under Low-View Settings

We propose a robust method for learning neural implicit functions that c...
12/23/2021

BANMo: Building Animatable 3D Neural Models from Many Casual Videos

Prior work for articulated 3D shape reconstruction often relies on speci...
04/17/2023

MoDA: Modeling Deformable 3D Objects from Casual Videos

In this paper, we focus on the challenges of modeling deformable 3D obje...

Code Repositories

nerfies

This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies.


view repo