Mixed Neural Voxels for Fast Multi-view Video Synthesis

12/01/2022
by   Feng Wang, et al.
0

Synthesizing high-fidelity videos from real-world multi-view input is challenging because of the complexities of real-world environments and highly dynamic motions. Previous works based on neural radiance fields have demonstrated high-quality reconstructions of dynamic scenes. However, training such models on real-world scenes is time-consuming, usually taking days or weeks. In this paper, we present a novel method named MixVoxels to better represent the dynamic scenes with fast training speed and competitive rendering qualities. The proposed MixVoxels represents the 4D dynamic scenes as a mixture of static and dynamic voxels and processes them with different networks. In this way, the computation of the required modalities for static voxels can be processed by a lightweight model, which essentially reduces the amount of computation, especially for many daily dynamic scenes dominated by the static background. To separate the two kinds of voxels, we propose a novel variation field to estimate the temporal variance of each voxel. For the dynamic voxels, we design an inner-product time query method to efficiently query multiple time steps, which is essential to recover the high-dynamic motions. As a result, with 15 minutes of training for dynamic scenes with inputs of 300-frame videos, MixVoxels achieves better PSNR than previous methods. Codes and trained models are available at https://github.com/fengres/mixvoxels

READ FULL TEXT

page 1

page 6

page 7

page 8

research
03/03/2021

Neural 3D Video Synthesis

We propose a novel approach for 3D video synthesis that is able to repre...
research
05/30/2022

Fast Dynamic Radiance Fields with Time-Aware Neural Voxels

Neural radiance fields (NeRF) have shown great success in modeling 3D sc...
research
05/30/2023

Template-free Articulated Neural Point Clouds for Reposable View Synthesis

Dynamic Neural Radiance Fields (NeRFs) achieve remarkable visual quality...
research
08/30/2021

Deep 3D Mask Volume for View Synthesis of Dynamic Scenes

Image view synthesis has seen great success in reconstructing photoreali...
research
05/31/2022

DeVRF: Fast Deformable Voxel Radiance Fields for Dynamic Scenes

Modeling dynamic scenes is important for many applications such as virtu...
research
05/24/2023

OD-NeRF: Efficient Training of On-the-Fly Dynamic Neural Radiance Fields

Dynamic neural radiance fields (dynamic NeRFs) have demonstrated impress...
research
12/02/2022

Fast Non-Rigid Radiance Fields from Monocularized Data

3D reconstruction and novel view synthesis of dynamic scenes from collec...

Please sign up or login with your details

Forgot password? Click here to reset