Future Transformer for Long-term Action Anticipation

05/27/2022
by   Dayoung Gong, et al.
0

The task of predicting future actions from a video is crucial for a real-world agent interacting with others. When anticipating actions in the distant future, we humans typically consider long-term relations over the whole sequence of actions, i.e., not only observed actions in the past but also potential actions in the future. In a similar spirit, we propose an end-to-end attention model for action anticipation, dubbed Future Transformer (FUTR), that leverages global attention over all input frames and output tokens to predict a minutes-long sequence of future actions. Unlike the previous autoregressive models, the proposed method learns to predict the whole sequence of future actions in parallel decoding, enabling more accurate and fast inference for long-term anticipation. We evaluate our method on two standard benchmarks for long-term action anticipation, Breakfast and 50 Salads, achieving state-of-the-art results.

READ FULL TEXT

page 7

page 8

page 17

page 18

research
10/20/2022

Rethinking Learning Approaches for Long-Term Action Anticipation

Action anticipation involves predicting future actions having observed t...
research
06/03/2021

Anticipative Video Transformer

We propose Anticipative Video Transformer (AVT), an end-to-end attention...
research
07/04/2023

Technical Report for Ego4D Long Term Action Anticipation Challenge 2023

In this report, we describe the technical details of our approach for th...
research
03/21/2022

LocATe: End-to-end Localization of Actions in 3D with Transformers

Understanding a person's behavior from their 3D motion is a fundamental ...
research
04/03/2018

When will you do what? - Anticipating Temporal Occurrences of Activities

Analyzing human actions in videos has gained increased attention recentl...
research
07/31/2023

AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?

Can we better anticipate an actor's future actions (e.g. mix eggs) by kn...
research
10/09/2018

Warped Hypertime Representations for Long-term Autonomy of Mobile Robots

This paper presents a novel method for introducing time into discrete an...

Please sign up or login with your details

Forgot password? Click here to reset