D^2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video

by   Tianhao Wu, et al.

Given a monocular video, segmenting and decoupling dynamic objects while recovering the static environment is a widely studied problem in machine intelligence. Existing solutions usually approach this problem in the image domain, limiting their performance and understanding of the environment. We introduce Decoupled Dynamic Neural Radiance Field (D^2NeRF), a self-supervised approach that takes a monocular video and learns a 3D scene representation which decouples moving objects, including their shadows, from the static background. Our method represents the moving objects and the static background by two separate neural radiance fields with only one allowing for temporal changes. A naive implementation of this approach leads to the dynamic component taking over the static one as the representation of the former is inherently more general and prone to overfitting. To this end, we propose a novel loss to promote correct separation of phenomena. We further propose a shadow field network to detect and decouple dynamically moving shadows. We introduce a new dataset containing various dynamic objects and shadows and demonstrate that our method can achieve better performance than state-of-the-art approaches in decoupling dynamic and static 3D objects, occlusion and shadow removal, and image segmentation for moving objects.


page 5

page 8

page 9

page 15

page 16

page 17

page 18

page 20


NeuralDiff: Segmenting 3D objects that move in egocentric videos

Given a raw video sequence taken from a freely-moving camera, we study t...

Attentional Separation-and-Aggregation Network for Self-supervised Depth-Pose Learning in Dynamic Scenes

Learning depth and ego-motion from unlabeled videos via self-supervision...

CeMNet: Self-supervised learning for accurate continuous ego-motion estimation

In this paper, we propose a novel self-supervised learning model for est...

Learned Equivariant Rendering without Transformation Supervision

We propose a self-supervised framework to learn scene representations fr...

Instance-aware multi-object self-supervision for monocular depth prediction

This paper proposes a self-supervised monocular image-to-depth predictio...

NeRF-DS: Neural Radiance Fields for Dynamic Specular Objects

Dynamic Neural Radiance Field (NeRF) is a powerful algorithm capable of ...

Static and Dynamic Concepts for Self-supervised Video Representation Learning

In this paper, we propose a novel learning scheme for self-supervised vi...

Please sign up or login with your details

Forgot password? Click here to reset