DeepAI
Log In Sign Up

D^2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video

05/31/2022
by   Tianhao Wu, et al.
0

Given a monocular video, segmenting and decoupling dynamic objects while recovering the static environment is a widely studied problem in machine intelligence. Existing solutions usually approach this problem in the image domain, limiting their performance and understanding of the environment. We introduce Decoupled Dynamic Neural Radiance Field (D^2NeRF), a self-supervised approach that takes a monocular video and learns a 3D scene representation which decouples moving objects, including their shadows, from the static background. Our method represents the moving objects and the static background by two separate neural radiance fields with only one allowing for temporal changes. A naive implementation of this approach leads to the dynamic component taking over the static one as the representation of the former is inherently more general and prone to overfitting. To this end, we propose a novel loss to promote correct separation of phenomena. We further propose a shadow field network to detect and decouple dynamically moving shadows. We introduce a new dataset containing various dynamic objects and shadows and demonstrate that our method can achieve better performance than state-of-the-art approaches in decoupling dynamic and static 3D objects, occlusion and shadow removal, and image segmentation for moving objects.

READ FULL TEXT

page 5

page 8

page 9

page 15

page 16

page 17

page 18

page 20

10/19/2021

NeuralDiff: Segmenting 3D objects that move in egocentric videos

Given a raw video sequence taken from a freely-moving camera, we study t...
11/18/2020

Attentional Separation-and-Aggregation Network for Self-supervised Depth-Pose Learning in Dynamic Scenes

Learning depth and ego-motion from unlabeled videos via self-supervision...
07/14/2020

Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance

Self-supervised monocular depth estimation presents a powerful method to...
11/11/2020

Learned Equivariant Rendering without Transformation Supervision

We propose a self-supervised framework to learn scene representations fr...
10/15/2022

Self-Improving SLAM in Dynamic Environments: Learning When to Mask

Visual SLAM – Simultaneous Localization and Mapping – in dynamic environ...
02/01/2021

Self-Supervised Equivariant Scene Synthesis from Video

We propose a self-supervised framework to learn scene representations fr...
07/26/2022

Static and Dynamic Concepts for Self-supervised Video Representation Learning

In this paper, we propose a novel learning scheme for self-supervised vi...