DeepAI AI Chat
Log In Sign Up

Deformable Neural Radiance Fields

by   Keunhong Park, et al.

We present the first method capable of photorealistically reconstructing a non-rigidly deforming scene using photos/videos captured casually from mobile phones. Our approach – D-NeRF – augments neural radiance fields (NeRF) by optimizing an additional continuous volumetric deformation field that warps each observed point into a canonical 5D NeRF. We observe that these NeRF-like deformation fields are prone to local minima, and propose a coarse-to-fine optimization method for coordinate-based models that allows for more robust optimization. By adapting principles from geometry processing and physical simulation to NeRF-like models, we propose an elastic regularization of the deformation field that further improves robustness. We show that D-NeRF can turn casually captured selfie photos/videos into deformable NeRF models that allow for photorealistic renderings of the subject from arbitrary viewpoints, which we dub "nerfies." We evaluate our method by collecting data using a rig with two mobile phones that take time-synchronized photos, yielding train/validation images of the same pose at different viewpoints. We show that our method faithfully reconstructs non-rigidly deforming scenes and reproduces unseen views with high fidelity.


page 1

page 4

page 5

page 6

page 7

page 9

page 10


NDF: Neural Deformable Fields for Dynamic Human Modelling

We propose Neural Deformable Fields (NDF), a new representation for dyna...

HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields

Neural Radiance Fields (NeRF) are able to reconstruct scenes with unprec...

DeVRF: Fast Deformable Voxel Radiance Fields for Dynamic Scenes

Modeling dynamic scenes is important for many applications such as virtu...

Deformable Model Driven Neural Rendering for High-fidelity 3D Reconstruction of Human Heads Under Low-View Settings

We propose a robust method for learning neural implicit functions that c...

BANMo: Building Animatable 3D Neural Models from Many Casual Videos

Prior work for articulated 3D shape reconstruction often relies on speci...

MoDA: Modeling Deformable 3D Objects from Casual Videos

In this paper, we focus on the challenges of modeling deformable 3D obje...

Code Repositories


This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies.

view repo