GaTector: A Unified Framework for Gaze Object Prediction

by   Binglu Wang, et al.

Gaze object prediction (GOP) is a newly proposed task that aims to discover the objects being stared at by humans. It is of great application significance but still lacks a unified solution framework. An intuitive solution is to incorporate an object detection branch into an existing gaze prediction method. However, previous gaze prediction methods usually use two different networks to extract features from scene image and head image, which would lead to heavy network architecture and prevent each branch from joint optimization. In this paper, we build a novel framework named GaTector to tackle the gaze object prediction problem in a unified way. Particularly, a specific-general-specific (SGS) feature extractor is firstly proposed to utilize a shared backbone to extract general features for both scene and head images. To better consider the specificity of inputs and tasks, SGS introduces two input-specific blocks before the shared backbone and three task-specific blocks after the shared backbone. Specifically, a novel defocus layer is designed to generate object-specific features for object detection task without losing information or requiring extra computations. Moreover, the energy aggregation loss is introduced to guide the gaze heatmap to concentrate on the stared box. In the end, we propose a novel mDAP metric that can reveal the difference between boxes even when they share no overlapping area. Extensive experiments on the GOO dataset verify the superiority of our method in all three tracks, i.e. object detection, gaze estimation, and gaze object prediction.


page 3

page 4

page 7


Joint Gaze-Location and Gaze-Object Detection

This paper proposes an efficient and effective method for joint gaze loc...

Object-aware Gaze Target Detection

Gaze target detection aims to predict the image location where the perso...

End-to-End Human-Gaze-Target Detection with Transformers

In this paper, we propose an effective and efficient method for Human-Ga...

GOO: A Dataset for Gaze Object Prediction in Retail Environments

One of the most fundamental and information-laden actions humans do is t...

Boosting Image-based Mutual Gaze Detection using Pseudo 3D Gaze

Mutual gaze detection, i.e., predicting whether or not two people are lo...

Gaze-based Object Detection in the Wild

In human-robot collaboration, one challenging task is to teach a robot n...

Free-View, 3D Gaze-Guided, Assistive Robotic System for Activities of Daily Living

Patients suffering from quadriplegia have limited body motion which prev...

Please sign up or login with your details

Forgot password? Click here to reset