DeepAI AI Chat
Log In Sign Up

ASM-Loc: Action-aware Segment Modeling for Weakly-Supervised Temporal Action Localization

by   Bo He, et al.
University of Maryland
Baidu, Inc.

Weakly-supervised temporal action localization aims to recognize and localize action segments in untrimmed videos given only video-level action labels for training. Without the boundary information of action segments, existing methods mostly rely on multiple instance learning (MIL), where the predictions of unlabeled instances (i.e., video snippets) are supervised by classifying labeled bags (i.e., untrimmed videos). However, this formulation typically treats snippets in a video as independent instances, ignoring the underlying temporal structures within and across action segments. To address this problem, we propose , a novel WTAL framework that enables explicit, action-aware segment modeling beyond standard MIL-based methods. Our framework entails three segment-centric components: (i) dynamic segment sampling for compensating the contribution of short actions; (ii) intra- and inter-segment attention for modeling action dynamics and capturing temporal dependencies; (iii) pseudo instance-level supervision for improving action boundary prediction. Furthermore, a multi-step refinement strategy is proposed to progressively improve action proposals along the model training process. Extensive experiments on THUMOS-14 and ActivityNet-v1.3 demonstrate the effectiveness of our approach, establishing new state of the art on both datasets. The code and models are publicly available at <>.


page 1

page 4

page 8

page 13


Weakly-Supervised Action Localization by Hierarchically-structured Latent Attention Modeling

Weakly-supervised action localization aims to recognize and localize act...

Proposal-Based Multiple Instance Learning for Weakly-Supervised Temporal Action Localization

Weakly-supervised temporal action localization aims to localize and reco...

Boosting Weakly-Supervised Temporal Action Localization with Text Information

Due to the lack of temporal annotation, current Weakly-supervised Tempor...

Weakly-Supervised Action Localization with Expectation-Maximization Multi-Instance Learning

Weakly-supervised action localization problem requires training a model ...

Enabling Weakly-Supervised Temporal Action Localization from On-Device Learning of the Video Stream

Detecting actions in videos have been widely applied in on-device applic...

Weakly-Supervised Dense Action Anticipation

Dense anticipation aims to forecast future actions and their durations f...

Weakly Supervised Temporal Action Localization with Segment-Level Labels

Temporal action localization presents a trade-off between test performan...