Log In Sign Up

Energy-Based Generative Cooperative Saliency Prediction

by   Jing Zhang, et al.

Conventional saliency prediction models typically learn a deterministic mapping from images to the corresponding ground truth saliency maps. In this paper, we study the saliency prediction problem from the perspective of generative models by learning a conditional probability distribution over saliency maps given an image, and treating the prediction as a sampling process. Specifically, we propose a generative cooperative saliency prediction framework based on the generative cooperative networks, where a conditional latent variable model and a conditional energy-based model are jointly trained to predict saliency in a cooperative manner. We call our model the SalCoopNets. The latent variable model serves as a fast but coarse predictor to efficiently produce an initial prediction, which is then refined by the iterative Langevin revision of the energy-based model that serves as a fine predictor. Such a coarse-to-fine cooperative saliency prediction strategy offers the best of both worlds. Moreover, we generalize our framework to the scenario of weakly supervised saliency prediction, where saliency annotation of training images is partially observed, by proposing a cooperative learning while recovering strategy. Lastly, we show that the learned energy function can serve as a refinement module that can refine the results of other pre-trained saliency prediction models. Experimental results show that our generative model can achieve state-of-the-art performance. Our code is publicly available at: <>.


page 7

page 10


Uncertainty Inspired RGB-D Saliency Detection

We propose the first stochastic framework to employ uncertainty for RGB-...

Weakly Supervised Top-down Salient Object Detection

Top-down saliency models produce a probability map that peaks at target ...

Concept Saliency Maps to Visualize Relevant Features in Deep Generative Models

Evaluating, explaining, and visualizing high-level concepts in generativ...

SaltiNet: Scan-path Prediction on 360 Degree Images using Saliency Volumes

We introduce SaltiNet, a deep neural network for scanpath prediction tra...

OpenSalicon: An Open Source Implementation of the Salicon Saliency Model

In this technical report, we present our publicly downloadable implement...

Temporal recurrences for video saliency prediction

This paper investigates modifying an existing neural network architectur...

Simple vs complex temporal recurrences for video saliency prediction

This paper investigates modifying an existing neural network architectur...