Saliency Prediction with External Knowledge

07/27/2020
by   Yifeng Zhang, et al.
0

The last decades have seen great progress in saliency prediction, with the success of deep neural networks that are able to encode high-level semantics. Yet, while humans have the innate capability in leveraging their knowledge to decide where to look (e.g. people pay more attention to familiar faces such as celebrities), saliency prediction models have only been trained with large eye-tracking datasets. This work proposes to bridge this gap by explicitly incorporating external knowledge for saliency models as humans do. We develop networks that learn to highlight regions by incorporating prior knowledge of semantic relationships, be it general or domain-specific, depending on the task of interest. At the core of the method is a new Graph Semantic Saliency Network (GraSSNet) that constructs a graph that encodes semantic relationships learned from external knowledge. A Spatial Graph Attention Network is then developed to update saliency features based on the learned graph. Experiments show that the proposed model learns to predict saliency from the external knowledge and outperforms the state-of-the-art on four saliency benchmarks.

READ FULL TEXT

page 1

page 6

page 8

research
11/14/2018

How Drones Look: Crowdsourced Knowledge Transfer for Aerial Video Saliency Prediction

In ground-level platforms, many saliency models have been developed to p...
research
08/04/2020

Implicit Saliency in Deep Neural Networks

In this paper, we show that existing recognition and localization deep a...
research
03/15/2018

What Catches the Eye? Visualizing and Understanding Deep Saliency Models

Deep convolutional neural networks have demonstrated high performances f...
research
10/08/2018

Saliency Prediction in the Deep Learning Era: An Empirical Investigation

Visual saliency models have enjoyed a big leap in performance in recent ...
research
05/02/2022

Understanding CNNs from excitations

For instance-level explanation, in order to reveal the relations between...
research
11/09/2015

Exploiting Egocentric Object Prior for 3D Saliency Detection

On a minute-to-minute basis people undergo numerous fluid interactions w...
research
07/22/2013

Visual saliency estimation by integrating features using multiple kernel learning

In the last few decades, significant achievements have been attained in ...

Please sign up or login with your details

Forgot password? Click here to reset