In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation

08/08/2022
by   Bolin Lai, et al.
6

In this paper, we present the first transformer-based model to address the challenging problem of egocentric gaze estimation. We observe that the connection between the global scene context and local visual information is vital for localizing the gaze fixation from egocentric video frames. To this end, we design the transformer encoder to embed the global context as one additional visual token and further propose a novel Global-Local Correlation (GLC) module to explicitly model the correlation of the global token and each local token. We validate our model on two egocentric video datasets - EGTEA Gaze+ and Ego4D. Our detailed ablation studies demonstrate the benefits of our method. In addition, our approach exceeds previous state-of-the-arts by a large margin. We also provide additional visualizations to support our claim that global-local correlation serves a key representation for predicting gaze fixation from egocentric videos. More details can be found in our website (https://bolinlai.github.io/GLC-EgoGazeEst).

READ FULL TEXT

page 1

page 2

page 4

page 9

page 18

page 19

page 20

page 21

research
06/06/2023

Human-Object Interaction Prediction in Videos through Gaze Following

Understanding the human-object interactions (HOIs) from a video is essen...
research
01/18/2021

LNSMM: Eye Gaze Estimation With Local Network Share Multiview Multitask

Eye gaze estimation has become increasingly significant in computer visi...
research
05/06/2023

Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipation

Egocentric gaze anticipation serves as a key building block for the emer...
research
09/22/2022

MGTR: End-to-End Mutual Gaze Detection with Transformer

People's looking at each other or mutual gaze is ubiquitous in our daily...
research
07/27/2018

Connecting Gaze, Scene, and Attention: Generalized Attention Estimation via Joint Modeling of Gaze and Scene Saliency

This paper addresses the challenging problem of estimating the general v...
research
03/30/2022

Collaborative Transformers for Grounded Situation Recognition

Grounded situation recognition is the task of predicting the main activi...
research
02/28/2019

Extended Gaze Following: Detecting Objects in Videos Beyond the Camera Field of View

In this paper we address the problems of detecting objects of interest i...

Please sign up or login with your details

Forgot password? Click here to reset