AOE-Net: Entities Interactions Modeling with Adaptive Attention Mechanism for Temporal Action Proposals Generation

10/05/2022
by   Khoa Vo, et al.
6

Temporal action proposal generation (TAPG) is a challenging task, which requires localizing action intervals in an untrimmed video. Intuitively, we as humans, perceive an action through the interactions between actors, relevant objects, and the surrounding environment. Despite the significant progress of TAPG, a vast majority of existing methods ignore the aforementioned principle of the human perceiving process by applying a backbone network into a given video as a black-box. In this paper, we propose to model these interactions with a multi-modal representation network, namely, Actors-Objects-Environment Interaction Network (AOE-Net). Our AOE-Net consists of two modules, i.e., perception-based multi-modal representation (PMR) and boundary-matching module (BMM). Additionally, we introduce adaptive attention mechanism (AAM) in PMR to focus only on main actors (or relevant objects) and model the relationships among them. PMR module represents each video snippet by a visual-linguistic feature, in which main actors and surrounding environment are represented by visual information, whereas relevant objects are depicted by linguistic features through an image-text model. BMM module processes the sequence of visual-linguistic features as its input and generates action proposals. Comprehensive experiments and extensive ablation studies on ActivityNet-1.3 and THUMOS-14 datasets show that our proposed AOE-Net outperforms previous state-of-the-art methods with remarkable performance and generalization for both TAPG and temporal action detection. To prove the robustness and effectiveness of AOE-Net, we further conduct an ablation study on egocentric videos, i.e. EPIC-KITCHENS 100 dataset. Source code is available upon acceptance.

READ FULL TEXT

page 3

page 8

page 22

page 23

page 24

page 25

research
10/21/2021

AEI: Actors-Environment Interaction with Adaptive Attention for Temporal Action Proposals Generation

Humans typically perceive the establishment of an action in a video thro...
research
03/16/2022

ABN: Agent-Aware Boundary Networks for Temporal Action Proposal Generation

Temporal action proposal generation (TAPG) aims to estimate temporal int...
research
12/12/2022

Contextual Explainable Video Representation: Human Perception-based Understanding

Video understanding is a growing field and a subject of intense research...
research
07/17/2021

Agent-Environment Network for Temporal Action Proposal Generation

Temporal action proposal generation is an essential and challenging task...
research
07/13/2018

Zoom-Net: Mining Deep Feature Interactions for Visual Relationship Recognition

Recognizing visual relationships <subject-predicate-object> among any pa...
research
06/21/2022

Pyramid Region-based Slot Attention Network for Temporal Action Proposal Generation

It has been found that temporal action proposal generation, which aims t...
research
07/05/2018

Detecting Visual Relationships Using Box Attention

In this paper we propose a new model for detecting visual relationships....

Please sign up or login with your details

Forgot password? Click here to reset