DeepAI AI Chat
Log In Sign Up

Boundary-aware Self-supervised Learning for Video Scene Segmentation

by   Jonghwan Mun, et al.
Kakao Corp.
HanYang University
Seoul National University

Self-supervised learning has drawn attention through its effectiveness in learning in-domain representations with no ground-truth annotations; in particular, it is shown that properly designed pretext tasks (e.g., contrastive prediction task) bring significant performance gains for downstream tasks (e.g., classification task). Inspired from this, we tackle video scene segmentation, which is a task of temporally localizing scene boundaries in a video, with a self-supervised learning framework where we mainly focus on designing effective pretext tasks. In our framework, we discover a pseudo-boundary from a sequence of shots by splitting it into two continuous, non-overlapping sub-sequences and leverage the pseudo-boundary to facilitate the pre-training. Based on this, we introduce three novel boundary-aware pretext tasks: 1) Shot-Scene Matching (SSM), 2) Contextual Group Matching (CGM) and 3) Pseudo-boundary Prediction (PP); SSM and CGM guide the model to maximize intra-scene similarity and inter-scene discrimination while PP encourages the model to identify transitional moments. Through comprehensive analysis, we empirically show that pre-training and transferring contextual representation are both critical to improving the video scene segmentation performance. Lastly, we achieve the new state-of-the-art on the MovieNet-SSeg benchmark. The code is available at


page 2

page 5

page 18

page 19

page 20


Unsupervised Object-Level Representation Learning from Scene Images

Contrastive self-supervised learning has largely narrowed the gap to sup...

Scene Consistency Representation Learning for Video Scene Segmentation

A long-term video, such as a movie or TV show, is composed of various sc...

Shot Contrastive Self-Supervised Learning for Scene Boundary Detection

Scenes play a crucial role in breaking the storyline of movies and TV ep...

Sentence Representation Learning with Generative Objective rather than Contrastive Objective

Though offering amazing contextualized token-level representations, curr...

Winning the CVPR'2021 Kinetics-GEBD Challenge: Contrastive Learning Approach

Generic Event Boundary Detection (GEBD) is a newly introduced task that ...

Self-supervised Pseudo-colorizing of Masked Cells

Self-supervised learning, which is strikingly referred to as the dark ma...

Delving into Inter-Image Invariance for Unsupervised Visual Representations

Contrastive learning has recently shown immense potential in unsupervise...

Code Repositories