Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling

05/26/2021
by   Akis Linardos, et al.
11

Since 2014 transfer learning has become the key driver for the improvement of spatial saliency prediction; however, with stagnant progress in the last 3-5 years. We conduct a large-scale transfer learning study which tests different ImageNet backbones, always using the same read out architecture and learning protocol adopted from DeepGaze II. By replacing the VGG19 backbone of DeepGaze II with ResNet50 features we improve the performance on saliency prediction from 78 backbones (such as EfficientNetB5) we observe no additional improvement on saliency prediction. By analyzing the backbones further, we find that generalization to other datasets differs substantially, with models being consistently overconfident in their fixation predictions. We show that by combining multiple backbones in a principled manner a good confidence calibration on unseen datasets can be achieved. This yields a significant leap in benchmark performance in and out-of-domain with a 15 percent point improvement over DeepGaze II to 93 on the MIT/Tuebingen Saliency Benchmark in all available metrics (AUC: 88.3 sAUC: 79.4

READ FULL TEXT

page 1

page 3

page 4

page 7

page 14

07/09/2020

n-Reference Transfer Learning for Saliency Prediction

Benefiting from deep learning research and large-scale datasets, salienc...
10/05/2016

DeepGaze II: Reading fixations from deep features trained on object recognition

Here we present DeepGaze II, a model that predicts where people look in ...
10/11/2018

Bottom-up Attention, Models of

In this review, we examine the recent progress in saliency prediction an...
02/24/2021

State-of-the-Art in Human Scanpath Prediction

The last years have seen a surge in models predicting the scanpaths of f...
08/25/2020

FastSal: a Computationally Efficient Network for Visual Saliency Prediction

This paper focuses on the problem of visual saliency prediction, predict...
03/11/2020

Unified Image and Video Saliency Modeling

Visual saliency modeling for images and videos is treated as two indepen...
09/26/2014

How close are we to understanding image-based saliency?

Within the set of the many complex factors driving gaze placement, the p...

Code Repositories

DeepGaze

pytorch implementation of the different DeepGaze models


view repo