iButter: Neural Interactive Bullet Time Generator for Human Free-viewpoint Rendering

08/12/2021
by   Liao Wang, et al.
4

Generating “bullet-time” effects of human free-viewpoint videos is critical for immersive visual effects and VR/AR experience. Recent neural advances still lack the controllable and interactive bullet-time design ability for human free-viewpoint rendering, especially under the real-time, dynamic and general setting for our trajectory-aware task. To fill this gap, in this paper we propose a neural interactive bullet-time generator (iButter) for photo-realistic human free-viewpoint rendering from dense RGB streams, which enables flexible and interactive design for human bullet-time visual effects. Our iButter approach consists of a real-time preview and design stage as well as a trajectory-aware refinement stage. During preview, we propose an interactive bullet-time design approach by extending the NeRF rendering to a real-time and dynamic setting and getting rid of the tedious per-scene training. To this end, our bullet-time design stage utilizes a hybrid training set, light-weight network design and an efficient silhouette-based sampling strategy. During refinement, we introduce an efficient trajectory-aware scheme within 20 minutes, which jointly encodes the spatial, temporal consistency and semantic cues along the designed trajectory, achieving photo-realistic bullet-time viewing experience of human activities. Extensive experiments demonstrate the effectiveness of our approach for convenient interactive bullet-time design and photo-realistic human free-viewpoint video generation.

READ FULL TEXT

page 1

page 3

page 5

page 6

page 7

page 8

page 12

research
04/30/2021

Editable Free-viewpoint Video Using a Layered Neural Representation

Generating free-viewpoint videos is critical for immersive VR/AR experie...
research
04/01/2021

Neural Video Portrait Relighting in Real-time via Consistency Modeling

Video portraits relighting is critical in user-facing human photography,...
research
06/28/2023

Envisioning a Next Generation Extended Reality Conferencing System with Efficient Photorealistic Human Rendering

Meeting online is becoming the new normal. Creating an immersive experie...
research
03/26/2023

Hardware Acceleration of Progressive Refinement Radiosity using Nvidia RTX

A vital component of photo-realistic image synthesis is the simulation o...
research
03/06/2017

Viewpoint Selection for Photographing Architectures

This paper studies the problem of how to choose good viewpoints for taki...
research
07/14/2022

Relighting4D: Neural Relightable Human from Videos

Human relighting is a highly desirable yet challenging task. Existing wo...
research
02/12/2022

NeuVV: Neural Volumetric Videos with Immersive Rendering and Editing

Some of the most exciting experiences that Metaverse promises to offer, ...

Please sign up or login with your details

Forgot password? Click here to reset