GliTr: Glimpse Transformers with Spatiotemporal Consistency for Online Action Prediction

10/24/2022
by   Samrudhdhi B Rangrej, et al.
0

Many online action prediction models observe complete frames to locate and attend to informative subregions in the frames called glimpses and recognize an ongoing action based on global and local information. However, in applications with constrained resources, an agent may not be able to observe the complete frame, yet must still locate useful glimpses to predict an incomplete action based on local information only. In this paper, we develop Glimpse Transformers (GliTr), which observe only narrow glimpses at all times, thus predicting an ongoing action and the following most informative glimpse location based on the partial spatiotemporal information collected so far. In the absence of a ground truth for the optimal glimpse locations for action recognition, we train GliTr using a novel spatiotemporal consistency objective: We require GliTr to attend to the glimpses with features similar to the corresponding complete frames (i.e. spatial consistency) and the resultant class logits at time t equivalent to the ones predicted using whole frames up to t (i.e. temporal consistency). Inclusion of our proposed consistency objective yields  10 the Something-Something-v2 (SSv2) dataset than the baseline cross-entropy objective. Overall, despite observing only  33 GliTr achieves 53.02 respectively.

READ FULL TEXT

page 3

page 5

page 6

page 13

research
04/01/2022

Consistency driven Sequential Transformers Attention Model for Partially Observable Scenes

Most hard attention models initially observe a complete scene to locate ...
research
04/19/2011

Hue Histograms to Spatiotemporal Local Features for Action Recognition

Despite the recent developments in spatiotemporal local features for act...
research
11/10/2015

Online Action Recognition based on Incremental Learning of Weighted Covariance Descriptors

Different from traditional action recognition based on video segments, o...
research
08/31/2020

Online Spatiotemporal Action Detection and Prediction via Causal Representations

In this thesis, we focus on video action understanding problems from an ...
research
11/15/2019

You Only Watch Once: A Unified CNN Architecture for Real-Time Spatiotemporal Action Localization

Spatiotemporal action localization requires incorporation of two sources...
research
06/09/2022

GateHUB: Gated History Unit with Background Suppression for Online Action Detection

Online action detection is the task of predicting the action as soon as ...

Please sign up or login with your details

Forgot password? Click here to reset