InterTracker: Discovering and Tracking General Objects Interacting with Hands in the Wild

by   Yanyan Shao, et al.

Understanding human interaction with objects is an important research topic for embodied Artificial Intelligence and identifying the objects that humans are interacting with is a primary problem for interaction understanding. Existing methods rely on frame-based detectors to locate interacting objects. However, this approach is subjected to heavy occlusions, background clutter, and distracting objects. To address the limitations, in this paper, we propose to leverage spatio-temporal information of hand-object interaction to track interactive objects under these challenging cases. Without prior knowledge of the general objects to be tracked like object tracking problems, we first utilize the spatial relation between hands and objects to adaptively discover the interacting objects from the scene. Second, the consistency and continuity of the appearance of objects between successive frames are exploited to track the objects. With this tracking formulation, our method also benefits from training on large-scale general object-tracking datasets. We further curate a video-level hand-object interaction dataset for testing and evaluation from 100DOH. The quantitative results demonstrate that our proposed method outperforms the state-of-the-art methods. Specifically, in scenes with continuous interaction with different objects, we achieve an impressive improvement of about 10 Our qualitative findings also illustrate that our method can produce more continuous trajectories for interacting objects.


page 1

page 6


Discovering A Variety of Objects in Spatio-Temporal Human-Object Interactions

Spatio-temporal Human-Object Interaction (ST-HOI) detection aims at dete...

Efficient Multiple Object Tracking Using Mutually Repulsive Active Membranes

Studies of social and group behavior in interacting organisms require hi...

Search Tracker: Human-derived object tracking in-the-wild through large-scale search and retrieval

Humans use context and scene knowledge to easily localize moving objects...

Exploiting Spatial Invariance for Scalable Unsupervised Object Tracking

The ability to detect and track objects in the visual world is a crucial...

Predicting Human Interaction via Relative Attention Model

Predicting human interaction is challenging as the on-going activity has...

Understanding and Exploiting Object Interaction Landscapes

Interactions play a key role in understanding objects and scenes, for bo...

Pixel personality for dense object tracking in a 2D honeybee hive

Tracking large numbers of densely-arranged, interacting objects is chall...

Please sign up or login with your details

Forgot password? Click here to reset