DeepAI AI Chat
Log In Sign Up

Looking Beyond Two Frames: End-to-End Multi-Object Tracking Using Spatial and Temporal Transformers

by   Tianyu Zhu, et al.
Monash University

Tracking a time-varying indefinite number of objects in a video sequence over time remains a challenge despite recent advances in the field. Ignoring long-term temporal information, most existing approaches are not able to properly handle multi-object tracking challenges such as occlusion. To address these shortcomings, we present MO3TR: a truly end-to-end Transformer-based online multi-object tracking (MOT) framework that learns to handle occlusions, track initiation and termination without the need for an explicit data association module or any heuristics/post-processing. MO3TR encodes object interactions into long-term temporal embeddings using a combination of spatial and temporal Transformers, and recursively uses the information jointly with the input data to estimate the states of all tracked objects over time. The spatial attention mechanism enables our framework to learn implicit representations between all the objects and the objects to the measurements, while the temporal attention mechanism focuses on specific parts of past information, allowing our approach to resolve occlusions over multiple frames. Our experiments demonstrate the potential of this new approach, reaching new state-of-the-art results on multiple MOT metrics for two popular multi-object tracking benchmarks. Our code will be made publicly available.


TrackFormer: Multi-Object Tracking with Transformers

We present TrackFormer, an end-to-end multi-object tracking and segmenta...

Standing Between Past and Future: Spatio-Temporal Modeling for Multi-Camera 3D Multi-Object Tracking

This work proposes an end-to-end multi-camera 3D multi-object tracking (...

Learning to Track with Object Permanence

Tracking by detection, the dominant approach for online multi-object tra...

Revealing Occlusions with 4D Neural Fields

For computer vision systems to operate in dynamic situations, they need ...

TubeTK: Adopting Tubes to Track Multi-Object in a One-Step Training Model

Multi-object tracking is a fundamental vision problem that has been stud...

Probabilistic Tracklet Scoring and Inpainting for Multiple Object Tracking

Despite the recent advances in multiple object tracking (MOT), achieved ...

Tracking The Untrackable: Learning To Track Multiple Cues with Long-Term Dependencies

The majority of existing solutions to the Multi-Target Tracking (MTT) pr...