DeepAI AI Chat
Log In Sign Up

InstanceFormer: An Online Video Instance Segmentation Framework

by   Rajat Koner, et al.
Universität München

Recent transformer-based offline video instance segmentation (VIS) approaches achieve encouraging results and significantly outperform online approaches. However, their reliance on the whole video and the immense computational complexity caused by full Spatio-temporal attention limit them in real-life applications such as processing lengthy videos. In this paper, we propose a single-stage transformer-based efficient online VIS framework named InstanceFormer, which is especially suitable for long and challenging videos. We propose three novel components to model short-term and long-term dependency and temporal coherence. First, we propagate the representation, location, and semantic information of prior instances to model short-term changes. Second, we propose a novel memory cross-attention in the decoder, which allows the network to look into earlier instances within a certain temporal window. Finally, we employ a temporal contrastive loss to impose coherence in the representation of an instance across all frames. Memory attention and temporal coherence are particularly beneficial to long-range dependency modeling, including challenging scenarios like occlusion. The proposed InstanceFormer outperforms previous online benchmark methods by a large margin across multiple datasets. Most importantly, InstanceFormer surpasses offline approaches for challenging and long datasets such as YouTube-VIS-2021 and OVIS. Code is available at


page 3

page 4

page 7

page 12

page 13


TALLFormer: Temporal Action Localization with Long-memory Transformer

Most modern approaches in temporal action localization divide this probl...

Robust Online Video Instance Segmentation with Track Queries

Recently, transformer-based methods have achieved impressive results on ...

Adaptive Human Matting for Dynamic Videos

The most recent efforts in video matting have focused on eliminating tri...

GRAtt-VIS: Gated Residual Attention for Auto Rectifying Video Instance Segmentation

Recent trends in Video Instance Segmentation (VIS) have seen a growing r...

Exploring Temporal Coherence for More General Video Face Forgery Detection

Although current face manipulation techniques achieve impressive perform...

MAIN: Multi-Attention Instance Network for Video Segmentation

Instance-level video segmentation requires a solid integration of spatia...

Memory-and-Anticipation Transformer for Online Action Understanding

Most existing forecasting systems are memory-based methods, which attemp...