Saliency Prediction with External Knowledge

07/27/2020
by   Yifeng Zhang, et al.
0

The last decades have seen great progress in saliency prediction, with the success of deep neural networks that are able to encode high-level semantics. Yet, while humans have the innate capability in leveraging their knowledge to decide where to look (e.g. people pay more attention to familiar faces such as celebrities), saliency prediction models have only been trained with large eye-tracking datasets. This work proposes to bridge this gap by explicitly incorporating external knowledge for saliency models as humans do. We develop networks that learn to highlight regions by incorporating prior knowledge of semantic relationships, be it general or domain-specific, depending on the task of interest. At the core of the method is a new Graph Semantic Saliency Network (GraSSNet) that constructs a graph that encodes semantic relationships learned from external knowledge. A Spatial Graph Attention Network is then developed to update saliency features based on the learned graph. Experiments show that the proposed model learns to predict saliency from the external knowledge and outperforms the state-of-the-art on four saliency benchmarks.

READ FULL TEXT

page 1

page 6

page 8

11/14/2018

How Drones Look: Crowdsourced Knowledge Transfer for Aerial Video Saliency Prediction

In ground-level platforms, many saliency models have been developed to p...
08/04/2020

Implicit Saliency in Deep Neural Networks

In this paper, we show that existing recognition and localization deep a...
03/15/2018

What Catches the Eye? Visualizing and Understanding Deep Saliency Models

Deep convolutional neural networks have demonstrated high performances f...
10/08/2018

Saliency Prediction in the Deep Learning Era: An Empirical Investigation

Visual saliency models have enjoyed a big leap in performance in recent ...
05/02/2022

Understanding CNNs from excitations

For instance-level explanation, in order to reveal the relations between...
10/11/2018

Bottom-up Attention, Models of

In this review, we examine the recent progress in saliency prediction an...
07/22/2013

Visual saliency estimation by integrating features using multiple kernel learning

In the last few decades, significant achievements have been attained in ...