Saliency Prediction with External Knowledge

by   Yifeng Zhang, et al.

The last decades have seen great progress in saliency prediction, with the success of deep neural networks that are able to encode high-level semantics. Yet, while humans have the innate capability in leveraging their knowledge to decide where to look (e.g. people pay more attention to familiar faces such as celebrities), saliency prediction models have only been trained with large eye-tracking datasets. This work proposes to bridge this gap by explicitly incorporating external knowledge for saliency models as humans do. We develop networks that learn to highlight regions by incorporating prior knowledge of semantic relationships, be it general or domain-specific, depending on the task of interest. At the core of the method is a new Graph Semantic Saliency Network (GraSSNet) that constructs a graph that encodes semantic relationships learned from external knowledge. A Spatial Graph Attention Network is then developed to update saliency features based on the learned graph. Experiments show that the proposed model learns to predict saliency from the external knowledge and outperforms the state-of-the-art on four saliency benchmarks.


page 1

page 6

page 8


How Drones Look: Crowdsourced Knowledge Transfer for Aerial Video Saliency Prediction

In ground-level platforms, many saliency models have been developed to p...

Implicit Saliency in Deep Neural Networks

In this paper, we show that existing recognition and localization deep a...

What Catches the Eye? Visualizing and Understanding Deep Saliency Models

Deep convolutional neural networks have demonstrated high performances f...

Saliency Prediction in the Deep Learning Era: An Empirical Investigation

Visual saliency models have enjoyed a big leap in performance in recent ...

Understanding CNNs from excitations

For instance-level explanation, in order to reveal the relations between...

Bottom-up Attention, Models of

In this review, we examine the recent progress in saliency prediction an...

Visual saliency estimation by integrating features using multiple kernel learning

In the last few decades, significant achievements have been attained in ...