Modelling Neighbor Relation in Joint Space-Time Graph for Video Correspondence Learning

09/28/2021
by   Zixu Zhao, et al.
0

This paper presents a self-supervised method for learning reliable visual correspondence from unlabeled videos. We formulate the correspondence as finding paths in a joint space-time graph, where nodes are grid patches sampled from frames, and are linked by two types of edges: (i) neighbor relations that determine the aggregation strength from intra-frame neighbors in space, and (ii) similarity relations that indicate the transition probability of inter-frame paths across time. Leveraging the cycle-consistency in videos, our contrastive learning objective discriminates dynamic objects from both their neighboring views and temporal views. Compared with prior works, our approach actively explores the neighbor relations of central instances to learn a latent association between center-neighbor pairs (e.g., "hand – arm") across time, thus improving the instance discrimination. Without fine-tuning, our learned representation outperforms the state-of-the-art self-supervised methods on a variety of visual tasks including video object propagation, part propagation, and pose keypoint tracking. Our self-supervised method also surpasses some fully supervised algorithms designed for the specific tasks.

READ FULL TEXT

page 1

page 6

page 8

research
12/09/2020

Contrastive Transformation for Self-supervised Correspondence Learning

In this paper, we focus on the self-supervised learning of visual corres...
research
03/18/2019

Learning Correspondence from the Cycle-Consistency of Time

We introduce a self-supervised method for learning visual correspondence...
research
06/25/2020

Space-Time Correspondence as a Contrastive Random Walk

This paper proposes a simple self-supervised approach for learning repre...
research
03/27/2022

Locality-Aware Inter-and Intra-Video Reconstruction for Self-Supervised Correspondence Learning

Our target is to learn visual correspondence from unlabeled videos. We d...
research
12/06/2022

Self-Supervised Correspondence Estimation via Multiview Registration

Video provides us with the spatio-temporal consistency needed for visual...
research
01/20/2022

Learning Pixel Trajectories with Multiscale Contrastive Random Walks

A range of video modeling tasks, from optical flow to multiple object tr...
research
03/30/2023

Soft Neighbors are Positive Supporters in Contrastive Visual Representation Learning

Contrastive learning methods train visual encoders by comparing views fr...

Please sign up or login with your details

Forgot password? Click here to reset