Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling

by   Akis Linardos, et al.

Since 2014 transfer learning has become the key driver for the improvement of spatial saliency prediction; however, with stagnant progress in the last 3-5 years. We conduct a large-scale transfer learning study which tests different ImageNet backbones, always using the same read out architecture and learning protocol adopted from DeepGaze II. By replacing the VGG19 backbone of DeepGaze II with ResNet50 features we improve the performance on saliency prediction from 78 backbones (such as EfficientNetB5) we observe no additional improvement on saliency prediction. By analyzing the backbones further, we find that generalization to other datasets differs substantially, with models being consistently overconfident in their fixation predictions. We show that by combining multiple backbones in a principled manner a good confidence calibration on unseen datasets can be achieved. This yields a significant leap in benchmark performance in and out-of-domain with a 15 percent point improvement over DeepGaze II to 93 on the MIT/Tuebingen Saliency Benchmark in all available metrics (AUC: 88.3 sAUC: 79.4


page 1

page 3

page 4

page 7

page 14


n-Reference Transfer Learning for Saliency Prediction

Benefiting from deep learning research and large-scale datasets, salienc...

DeepGaze II: Reading fixations from deep features trained on object recognition

Here we present DeepGaze II, a model that predicts where people look in ...

Bottom-up Attention, Models of

In this review, we examine the recent progress in saliency prediction an...

State-of-the-Art in Human Scanpath Prediction

The last years have seen a surge in models predicting the scanpaths of f...

FastSal: a Computationally Efficient Network for Visual Saliency Prediction

This paper focuses on the problem of visual saliency prediction, predict...

Unified Image and Video Saliency Modeling

Visual saliency modeling for images and videos is treated as two indepen...

How close are we to understanding image-based saliency?

Within the set of the many complex factors driving gaze placement, the p...

Code Repositories


pytorch implementation of the different DeepGaze models

view repo