The Emergence of Objectness: Learning Zero-Shot Segmentation from Videos

by   Runtao Liu, et al.

Humans can easily segment moving objects without knowing what they are. That objectness could emerge from continuous visual observations motivates us to model grouping and movement concurrently from unlabeled videos. Our premise is that a video has different views of the same scene related by moving components, and the right region segmentation and region flow would allow mutual view synthesis which can be checked from the data itself without any external supervision. Our model starts with two separate pathways: an appearance pathway that outputs feature-based region segmentation for a single image, and a motion pathway that outputs motion features for a pair of images. It then binds them in a conjoint representation called segment flow that pools flow offsets over each region and provides a gross characterization of moving regions for the entire scene. By training the model to minimize view synthesis errors based on segment flow, our appearance and motion pathways learn region segmentation and flow estimation automatically without building them up from low-level edges or optical flows respectively. Our model demonstrates the surprising emergence of objectness in the appearance pathway, surpassing prior works on zero-shot object segmentation from an image, moving object segmentation from a video with unsupervised test-time adaptation, and semantic image segmentation by supervised fine-tuning. Our work is the first truly end-to-end zero-shot object segmentation from videos. It not only develops generic objectness for segmentation and tracking, but also outperforms prevalent image-based contrastive learning methods without augmentation engineering.


page 2

page 4

page 6

page 8

page 9

page 16

page 17


Bootstrapping Objectness from Videos by Relaxed Common Fate and Visual Grouping

We study learning object segmentation from unlabeled videos. Humans can ...

Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motion

Motion, measured via optical flow, provides a powerful cue to discover a...

NudgeSeg: Zero-Shot Object Segmentation by Repeated Physical Interaction

Recent advances in object segmentation have demonstrated that deep neura...

Learning Video Object Segmentation from Unlabeled Videos

We propose a new method for video object segmentation (VOS) that address...

Optical Flow with Semantic Segmentation and Localized Layers

Existing optical flow methods make generic, spatially homogeneous, assum...

Divided Attention: Unsupervised Multi-Object Discovery with Contextually Separated Slots

We introduce a method to segment the visual field into independently mov...

Motion-Attentive Transition for Zero-Shot Video Object Segmentation

In this paper, we present a novel Motion-Attentive Transition Network (M...

Please sign up or login with your details

Forgot password? Click here to reset