INV: Towards Streaming Incremental Neural Videos

02/03/2023
by   Shengze Wang, et al.
0

Recent works in spatiotemporal radiance fields can produce photorealistic free-viewpoint videos. However, they are inherently unsuitable for interactive streaming scenarios (e.g. video conferencing, telepresence) because have an inevitable lag even if the training is instantaneous. This is because these approaches consume videos and thus have to buffer chunks of frames (often seconds) before processing. In this work, we take a step towards interactive streaming via a frame-by-frame approach naturally free of lag. Conventional wisdom believes that per-frame NeRFs are impractical due to prohibitive training costs and storage. We break this belief by introducing Incremental Neural Videos (INV), a per-frame NeRF that is efficiently trained and streamable. We designed INV based on two insights: (1) Our main finding is that MLPs naturally partition themselves into Structure and Color Layers, which store structural and color/texture information respectively. (2) We leverage this property to retain and improve upon knowledge from previous frames, thus amortizing training across frames and reducing redundant learning. As a result, with negligible changes to NeRF, INV can achieve good qualities (>28.6db) in 8min/frame. It can also outperform prior SOTA in 19 Additionally, our Temporal Weight Compression reduces the per-frame size to 0.3MB/frame (6.6 is naturally fit for streaming. While this work does not achieve real-time training, it shows that incremental approaches like INV present new possibilities in interactive 3D streaming. Moreover, our discovery of natural information partition leads to a better understanding and manipulation of MLPs. Code and dataset will be released soon.

READ FULL TEXT

page 1

page 4

page 5

page 6

page 8

page 9

research
10/26/2022

Streaming Radiance Fields for 3D Video Synthesis

We present an explicit-grid based method for efficiently reconstructing ...
research
03/17/2023

MoRF: Mobile Realistic Fullbody Avatars from a Monocular Video

We present a new approach for learning Mobile Realistic Fullbody (MoRF) ...
research
12/14/2017

Learning Binary Residual Representations for Domain-specific Video Streaming

We study domain-specific video streaming. Specifically, we target a stre...
research
03/22/2016

Stitching Stabilizer: Two-frame-stitching Video Stabilization for Embedded Systems

In conventional electronic video stabilization, the stabilized frame is ...
research
10/06/2020

Online Action Detection in Streaming Videos with Time Buffers

We formulate the problem of online temporal action detection in live str...
research
05/24/2021

VAD360: Viewport Aware Dynamic 360-Degree Video Frame Tiling

360 videos a.k.a. spherical videos are getting popular among users never...
research
11/02/2022

Learning a Condensed Frame for Memory-Efficient Video Class-Incremental Learning

Recent incremental learning for action recognition usually stores repres...

Please sign up or login with your details

Forgot password? Click here to reset