Layered Neural Rendering for Retiming People in Video

09/16/2020
by   Erika Lu, et al.
10

We present a method for retiming people in an ordinary, natural video—manipulating and editing the time in which different motions of individuals in the video occur. We can temporally align different motions, change the speed of certain actions (speeding up/slowing down, or entirely "freezing" people), or "erase" selected people from the video altogether. We achieve these effects computationally via a dedicated learning-based layered video representation, where each frame in the video is decomposed into separate RGBA layers, representing the appearance of different people in the video. A key property of our model is that it not only disentangles the direct motions of each person in the input video, but also correlates each person automatically with the scene changes they generate—e.g., shadows, reflections, and motion of loose clothing. The layers can be individually retimed and recombined into a new video, allowing us to achieve realistic, high-quality renderings of retiming effects for real-world videos depicting complex actions and involving multiple individuals, including dancing, trampoline jumping, or group running.

READ FULL TEXT

page 2

page 3

page 4

page 6

page 7

page 10

page 12

page 13

research
09/23/2021

Layered Neural Atlases for Consistent Video Editing

We present a method that decomposes, or "unwraps", an input video into a...
research
04/10/2021

Do as we do: Multiple Person Video-To-Video Transfer

Our goal is to transfer the motion of real people from a source video to...
research
11/25/2019

Appearance Composing GAN: A General Method for Appearance-Controllable Human Video Motion Transfer

Due to the rapid development of GANs, there has been significant progres...
research
12/04/2018

The Visual Centrifuge: Model-Free Layered Video Representations

True video understanding requires making sense of non-lambertian scenes ...
research
04/30/2021

Editable Free-viewpoint Video Using a Layered Neural Representation

Generating free-viewpoint videos is critical for immersive VR/AR experie...
research
04/21/2022

Unsupervised Video Interpolation by Learning Multilayered 2.5D Motion Fields

The problem of video frame interpolation is to increase the temporal res...
research
05/14/2021

Omnimatte: Associating Objects and Their Effects in Video

Computer vision is increasingly effective at segmenting objects in image...

Please sign up or login with your details

Forgot password? Click here to reset