Omnimatte: Associating Objects and Their Effects in Video

05/14/2021
by   Erika Lu, et al.
8

Computer vision is increasingly effective at segmenting objects in images and videos; however, scene effects related to the objects—shadows, reflections, generated smoke, etc—are typically overlooked. Identifying such scene effects and associating them with the objects producing them is important for improving our fundamental understanding of visual scenes, and can also assist a variety of applications such as removing, duplicating, or enhancing objects in video. In this work, we take a step towards solving this novel problem of automatically associating objects with their effects in video. Given an ordinary video and a rough segmentation mask over time of one or more subjects of interest, we estimate an omnimatte for each subject—an alpha matte and color image that includes the subject along with all its related time-varying scene elements. Our model is trained only on the input video in a self-supervised manner, without any manual labels, and is generic—it produces omnimattes automatically for arbitrary objects and a variety of effects. We show results on real-world videos containing interactions between different types of subjects (cars, animals, people) and complex effects, ranging from semi-transparent elements such as smoke and reflections, to fully opaque effects such as objects attached to the subject.

READ FULL TEXT

page 1

page 2

page 3

page 4

page 6

page 7

page 8

page 9

research
04/13/2020

SpeedNet: Learning the Speediness in Videos

We wish to automatically predict the "speediness" of moving objects in v...
research
08/02/2021

Consistent Depth of Moving Objects in Video

We present a method to estimate depth of a dynamic scene, containing arb...
research
12/13/2017

Self-Supervised Depth Learning for Urban Scene Understanding

As an agent moves through the world, the apparent motion of scene elemen...
research
09/14/2023

OmnimatteRF: Robust Omnimatte with 3D Background Modeling

Video matting has broad applications, from adding interesting effects to...
research
03/03/2022

Self-supervised Transparent Liquid Segmentation for Robotic Pouring

Liquid state estimation is important for robotics tasks such as pouring;...
research
09/16/2020

Layered Neural Rendering for Retiming People in Video

We present a method for retiming people in an ordinary, natural video—ma...
research
09/25/2017

Realizing Half-Diminished Reality from Video Stream of Manipulating Objects

When we watch a video, in which human hands manipulate objects, these ha...

Please sign up or login with your details

Forgot password? Click here to reset