DeepAI AI Chat
Log In Sign Up

GaTector: A Unified Framework for Gaze Object Prediction

by   Binglu Wang, et al.

Gaze object prediction (GOP) is a newly proposed task that aims to discover the objects being stared at by humans. It is of great application significance but still lacks a unified solution framework. An intuitive solution is to incorporate an object detection branch into an existing gaze prediction method. However, previous gaze prediction methods usually use two different networks to extract features from scene image and head image, which would lead to heavy network architecture and prevent each branch from joint optimization. In this paper, we build a novel framework named GaTector to tackle the gaze object prediction problem in a unified way. Particularly, a specific-general-specific (SGS) feature extractor is firstly proposed to utilize a shared backbone to extract general features for both scene and head images. To better consider the specificity of inputs and tasks, SGS introduces two input-specific blocks before the shared backbone and three task-specific blocks after the shared backbone. Specifically, a novel defocus layer is designed to generate object-specific features for object detection task without losing information or requiring extra computations. Moreover, the energy aggregation loss is introduced to guide the gaze heatmap to concentrate on the stared box. In the end, we propose a novel mDAP metric that can reveal the difference between boxes even when they share no overlapping area. Extensive experiments on the GOO dataset verify the superiority of our method in all three tracks, i.e. object detection, gaze estimation, and gaze object prediction.


page 3

page 4

page 7


End-to-End Human-Gaze-Target Detection with Transformers

In this paper, we propose an effective and efficient method for Human-Ga...

GOO: A Dataset for Gaze Object Prediction in Retail Environments

One of the most fundamental and information-laden actions humans do is t...

Boosting Image-based Mutual Gaze Detection using Pseudo 3D Gaze

Mutual gaze detection, i.e., predicting whether or not two people are lo...

Gaze-based Object Detection in the Wild

In human-robot collaboration, one challenging task is to teach a robot n...

Spot the Difference by Object Detection

In this paper, we propose a simple yet effective solution to a change de...

Patch-level Gaze Distribution Prediction for Gaze Following

Gaze following aims to predict where a person is looking in a scene, by ...