Log In Sign Up

ST-HOI: A Spatial-Temporal Baseline for Human-Object Interaction Detection in Videos

by   Meng-Jiun Chiou, et al.

Detecting human-object interactions (HOI) is an important step toward a comprehensive visual understanding of machines. While detecting non-temporal HOIs (e.g., sitting on a chair) from static images is feasible, it is unlikely even for humans to guess temporal-related HOIs (e.g., opening/closing a door) from a single video frame, where the neighboring frames play an essential role. However, conventional HOI methods operating on only static images have been used to predict temporal-related interactions, which is essentially guessing without temporal contexts and may lead to sub-optimal performance. In this paper, we bridge this gap by detecting video-based HOIs with explicit temporal information. We first show that a naive temporal-aware variant of a common action detection baseline does not work on video-based HOIs due to a feature-inconsistency issue. We then propose a simple yet effective architecture named Spatial-Temporal HOI Detection (ST-HOI) utilizing temporal information such as human and object trajectories, correctly-localized visual features, and spatial-temporal masking pose features. We construct a new video HOI benchmark dubbed VidHOI where our proposed approach serves as a solid baseline.


page 1

page 4

page 8


Spatio-Temporal Context for Action Detection

Research in action detection has grown in the recentyears, as it plays a...

Extracting temporal features into a spatial domain using autoencoders for sperm video analysis

In this paper, we present a two-step deep learning method that is used t...

Object Referring in Videos with Language and Human Gaze

We investigate the problem of object referring (OR) i.e. to localize a t...

Towards Spatio-Temporal Video Scene Text Detection via Temporal Clustering

With only bounding-box annotations in the spatial domain, existing video...

DeepSignals: Predicting Intent of Drivers Through Visual Signals

Detecting the intention of drivers is an essential task in self-driving,...

Visual Exploration of Movement Relatedness for Multi-species Ecology Analysis

Advances in GPS telemetry technology have enabled analysis of animal mov...

Code Repositories


Official implementation of "ST-HOI: A Spatial-Temporal Baseline for Human-Object Interaction Detection in Videos" (ACM ICMRW 2021)

view repo