Photorealistic Style Transfer with Screened Poisson Equation
Have you ever taken a picture only to find out that an unimportant background object ended up being overly salient? Or one of those team sports photos where your favorite player blends with the rest? Wouldn't it be nice if you could tweak these pictures just a little bit so that the distractor would be attenuated and your favorite player will stand-out among her peers? Manipulating images in order to control the saliency of objects is the goal of this paper. We propose an approach that considers the internal color and saliency properties of the image. It changes the saliency map via an optimization framework that relies on patch-based manipulation using only patches from within the same image to achieve realistic looking results. Applications include object enhancement, distractors attenuation and background decluttering. Comparing our method to previous ones shows significant improvement, both in the achieved saliency manipulation and in the realistic appearance of the resulting images.READ FULL TEXT VIEW PDF
Photorealistic Style Transfer with Screened Poisson Equation
Saliency detection, the task of identifying the salient and non-salient regions of an image, has drawn considerable amount of research in recent years, e.g., [14, 18, 22, 33, 35]. Our interest is in manipulating an image in order to modify its corresponding saliency map. This task has been named before as attention retargeting  or re-attentionizing  and has not been explored much, even though it could be useful for various applications such as object enhancement [25, 27], directing viewer’s attention in mixed reality  or in computer games , distractor removal , background de-emphasis  and improving image aesthetics [15, 31, 34]. Imagine being able to highlight your child who stands in the chorus line, or making it easier for a person with a visual impairment to find an object by making it more salient. Such manipulations are the aim of this paper.
Image editors use complex manipulations to enhance a particular object in a photo. They combine effects such as increasing the object’s exposure, decreasing the background exposure, changing hue, increasing saturation, or blurring the background. More importantly, they adapt the manipulation to each photo – if the object is too dark they increase its exposure, if its colors are too flat they increase its saturation etc. Such complex manipulations are difficult for novice users that often do not know what to change and how. Instead, we provide the non-experts an intuitive way to highlight objects. All they need to do is mark the target region and tune a single parameter, that is directly linked to the desired saliency contrast between the target region and the rest of the image. An example manipulation is presented in Figure 1.
The approach we propose makes four key contributions over previous solutions. First, our approach handles multiple image regions and can either increase or decrease the saliency of each region. This is essential in many cases to achieve the desired enhancement effect. Second, we produce realistic and natural looking results by manipulating the image in a way that is consistent with its internal characteristics. This is different from many previous methods that enhance a region by recoloring it with a preeminent color that is often very non-realistic (e.g., turning leaves to cyan and goats to purple). Third, our approach provides the user with an intuitive way for controlling the level of enhancement. This important feature is completely missing from all previous methods. Last, but not least, we present the first benchmark for object enhancement that consists of over 650 images. This is at least an order of magnitude larger than the test-sets of previous works, that were satisfied with testing on a very small number of cherry-picked images.
The algorithm we propose aims at globally optimizing an overall objective that considers the image saliency map. A key component to our solution is replacing properties of image patches in the target regions with other patches from the same image. This concept is a key ingredient in many patch-bases synthesis and analysis methods, such as texture synthesis , image completion , highlighting irregularities , image summarization , image compositing and harmonization  and recently highlighting non-local variations . Our method follows this line of work as we replace patches in the target regions with similar ones from other image regions. Differently from those methods, our patch-to-patch similarity considers the saliency of the patches with respect to the rest of the image. This is necessary to optimize the saliency-based objective we propose. A key observation we make is that these patch replacements do not merely copy the saliency of the source patch to the target location as saliency is a complex global phenomena (similar idea was suggested in 
for saliency detection). Instead, we interleave saliency estimation within the patch synthesis process. In addition, we do not limit the editing to the target region but rather change (if necessary) the entire image to obtain the desired global saliency goal.
we propose a new quantitative criteria to assess performance of saliency editing algorithms by comparing two properties to previous methods: (i) The ability to manipulate an image such that the saliency map of the result matches the user goal. (ii) The realism of the manipulated image. These properties are evaluated via qualitative means, quantitative measures and user studies. Our experiments show a significant improvement over previous methods. We further show that our general framework is applicable to two other applications: distractor attenuation and background decluttering.
Attention retargeting methods have a mutual goal – to enhance a selected region. They differ, however, in the way the image is manipulated [15, 25, 27, 30, 31]. We next briefly describe the key ideas behind these methods. A more thorough review and comparison is provided in .
Some approaches are based solely on color manipulation [25, 27]. This usually suffices to enhance the object of interest, but often results in non-realistic manipulations, such as purple snakes or blue flamingos. Approaches that integrate also other saliency cues, such as saturation, illumination and sharpness have also been proposed [15, 26, 30, 31]. While attempting to produce realistic and aesthetic results, they do not always succeed, as we show empirically later on.
Recently Yan et al.  suggested a deep convolutional network to learn transformations that adjust image aesthetics. One of the effects they study is Foreground Pop-Out, which is similar in spirit to object saliency enhancement. Their method produces aesthetic results, however, it requires intensive manual labeling by professional artists in the training phase and it is limited to the labeled effect used by the professional.
Our Object Enhancement formulation takes as input an image , a target region mask and the desired saliency contrast between the target region and the rest of the image. It generates a manipulated image whose corresponding saliency map is denoted by .
We pose this task as a patch-based optimization problem over the image . The objective we define distinguishes between salient and non-salient patches and pushes for manipulation that matches the saliency contrast . To do this we extract from the input image two databases of patches of size : of patches with high saliency and of patches with low saliency. The thresholds and are found via our optimization (explained below).
To increase the saliency of patches and decrease the saliency of patches we define the following energy function:
where is the sum of squared distances (SSD) over color channels between patches and . The role of the third term, , is to preserve the gradients of the original image . The balance between the color channels and the gradient channels is controlled by .
Recall, that our goal in minimizing (1) is to generate an image with saliency map , such that the contrast in saliency between and the rest of the image is . The key to this lies in the construction of the patch sets and . The higher the threshold the more salient will be the patches in and in return those in . Similarly, the lower the threshold the less salient will be the patches in and in return those outside of . Our algorithm performs an approximate greedy search over the thresholds to determine their values.
To formulate mathematically the affect of the user control parameter we further define a function that computes the saliency difference between pixels in the target region and those outside it:
and seek to minimize the saliency-based energy term:
For robustness to outliers we only consider the() most salient pixels in and outside in the mean calculation.
The optimization problem in (1) is non-convex with respect to the databases , . To solve it, we perform an approximate greedy search over the thresholds , to determine their values. Given a choice of threshold values, we construct the corresponding databases and then minimize the objective in (1) w.r.t. , while keeping the databases fixed. Pseudo-code is provided in Algorithm 1.
Image Update: Manipulate to enhance the region . Patches are replaced with similar ones from , while, patches are replaced with similar ones from .
Database Update: Reassign the patches from the input image into two databases, and , of salient and non-salient patches, respectively. The databases are updated at every iteration by shifting the thresholds , in order to find values that yield the desired foreground enhancement and background demotion effects (according to ).
Fine-scale Refinement: We observed that updating both the image and the databases , at all scales, does not contribute much to the results, as most changes happen already at coarse scales. Similar behavior was observed by  in retargeting and by  in reshuffling. Hence, the iterations of updating the image and databases are performed only at coarse resolution. After convergence, we continue and apply the Image Update step at finer scales, while the databases are held fixed. Between scales, we down-sample the input image to be of the same size as , and then reassign the patches from the scaled into and using the current thresholds.
In our implementation we use a Gaussian pyramid with 0.5 scale gaps, and apply 5-20 iterations, more at coarse scales and less at fine scales. The coarsest scale is set to be 150 pixels width.
Throughout the algorithm when a saliency map is computed for either or we use a modification of . Because we want the saliency map to be as sharp as possible, we use a small patch size of . In addition, we omit the center prior which assumes higher saliency for patches at the center of the image. We found it to ambiguate the differences in saliency between patches, which might be good when comparing prediction results to smoothed ground-truth maps, but not for our purposes. We selected the saliency estimation of  since its core is to find what makes a patch distinct. It assigns a score to each patch based on the inner statistics of the patches in the image, which is a beneficial property to our method.
In this step we minimize (1) with respect to , while holding the databases fixed. This resembles the optimization proposed by  for image synthesis. It differs, however, in two important ways. First,  consider only luminance gradients, while we consider gradients of all three color channels. This improves the smoothness of the color manipulation, preventing generation of spurious color edges, like those evident in Figure 1(c). It guides the optimization to abide to the color gradients of the original image and often leads to improved results (Figure 1(d)).
As was shown in , the energy terms in (1) can be optimized by combining a patch search-and-vote scheme and a discrete Screened Poisson equation that was originally suggested by  for gradient domain problems. At each scale, every iteration starts with a search-and-vote scheme that replaces patches of color with similar ones from the appropriate patch database. For each patch we search for the Nearest Neighbor patch . Note, that we perform two separate searches, for the target region in and for the background in . This is the second difference from  where a single search is performed over one source region.
To reduce computation time the databases are represented as two images: and . The search is performed using PatchMatch  with patch size and translation transformation only (we found that rotation and scale were not beneficial). In the vote step, every target pixel is assigned the mean color of all the patches that overlap with it. The voted color image is then combined with the original gradients of image using a Screened Poisson solver to obtain the final colors of that iteration. We fixed as the gradients weight.
Having constructed a new image , we compute its saliency map to be used in the database update step explained next.
The purpose of the database update step is to search for the appropriate thresholds that split the patches of into salient and non-salient databases. Our underlying assumption is that there exist threshold values that result in minimizing the objective of (3).
Recall that the databases are constructed using two thresholds on the saliency map such that and . An exhaustive search over all possible threshold values is non-tractable. Instead, we perform an approximate search that starts from a low value for and a high value for and then gradually increases the first and reduces the second until satisfactory values are found. Note, that and could be overlapping if .
The naive thresholds , would leave only the most salient patches in and the most non-salient in . This, however, could lead to non-realistic results and might not match the user’s input for a specific saliency contrast . To find a solution which considers realism and the user’s input we seek the maximal and minimal that minimize the saliency term .
At each iteration we continue the search over the thresholds by gradually updating them:
where is the inverse of the target region . Since the values of the thresholds are not bounded, we trim them to be in the range of . Convergence is declared when , i.e., when the desired contrast is reached. If convergence fails the iterations are stopped when the thresholds stop changing between subsequent iterations. In our implementation and .
An important property of our method is that if (or very high) and (or very low) the image would be left unchanged as the solution where all patches are replaced by themselves will lead to a zero error of our objective energy function (1).
The only parameter we request the user to provide is which determines the enhancement level. We argue that this parameter is easy and intuitive to tune as it directly relates to the desired saliency contrast between the target region and the background. We used a default value of , for which convergence was achieved for of the images. In only a few cases the result was not aesthetically pleasing and we used other values in the range . Throughout the paper, if not mentioned otherwise, .
An additional parameter is , which was fixed to in our implementation. In practice, we found that for any value we got approximately the same results, while for the manipulated images tend to be blurry (mathematical analysis can be found in , since our is equivalent to that of the screened Poisson).
Our algorithm is not guaranteed to reach a global minima. However we found that typically the manipulated image is visually plausible, and pertains a good match to the desired saliency.
It takes around 2 minutes to run our algorithm on a image – the most time demanding step of our method is solving the screened Poisson equation at each iteration. Since our main focus was on quality we did not optimize the implementation for speed. Significant speed-up could be achieved by adopting the method of . As was shown by  replacing these fast pyramidal convolutions with our current solver, will reduce run-time from minutes to several seconds.
To evaluate object enhancement one must consider two properties of the manipulated image: (i) the similarity of its saliency map to the user-provided target, and, (ii) whether it looks realistic. Through these two properties we compare our algorithm to HAG , OHA , and WSR , that were identified as top performers in . 111Code for WSR and HAG is not publicly available, hence we used our own implementation that led to similar results on examples from their papers. We will make this code publicly available for future comparisons. For OHA we used the original code. .
We start by providing a qualitative sense of what our algorithm can achieve in Figure 9. Many more results are provided in the supplementary, and we encourage the reader to view them. Comparing to OHA, it is evident that our results are more realistic. OHA changes the hue of the selected object such that its new color is unique with respect to the color histogram of the rest of the image. This often results in unrealistic colors. The results of WSR and HAG, on the other hand, are typically realistic since their manipulation is restricted not to deviate too much from the original image in order to achieve realistic outcomes. This, however, comes at the expense of often failing to achieve the desired object enhancement altogether.
The ability of our approach to simultaneously reduce and increase saliency of different regions is essential in some cases, e.g. Figure 9, rows 1 and 4. In addition, it is important to note that our manipulation latches onto the internal statistics of the image and emphasizes the objects via a combination of different saliency cues, such as color, saturation and illumination. Examples of these complex effects are presented in Figure 9, rows 2, 6 and 7, respectively.
A new benchmark: To perform quantitative evaluation we built a corpus of images gathered from previous papers on object enhancement and saliency [2, 8, 13, 16, 20, 25] as well as images from MS COCO . Our dataset is the largest ever built and tested for this task and sets a new benchmark in this area. We will make our dataset, code and results publicly available.
Enhancement evaluation: To measure how successful a manipulated image is, we do the following. We take the user provided mask as the ground-truth saliency map. We then compute the saliency map of the manipulated image and compare it to the ground-truth. To provide a reliable assessment we use five different salient object detection methods: MBS , HSL , DSR , PCA  and MDP, each based on different principles (patch based, CNN, geodesic distance etc.). The computed saliency maps are compared to the ground-truth using two commonly-used metrics for saliency evaluation: (i) Pearson’s-Correlation-Coefficient (CC) which was recommended by  as the best option for assessing saliency maps, and, (ii) Weighted F-beta (WFB)  which was shown to be a preferred choice for evaluation of foreground maps.
The bar plots in Figure 3 show that the saliency maps of our manipulated images are more similar to the ground-truth than those of OHA, WSR and HAG. This is true for both saliency measures and for all five methods for saliency estimation.
Realism: As mentioned earlier, being able to enhance a region does not suffice. We must also verify that the manipulated images look plausible and realistic. We measure this via a user survey. Each image was presented to human participants who were asked a simple question: “Does the image look realistic?” The scores were given on a scale of [1-9], where 9 is ’definitely realistic’ and 1 is ’definitely unrealistic’. We used Amazon Mechanical Turk to collect 20 annotations per image, where each worker viewed only one version of each image out of five. The survey was performed on a random subset of 20% of the data-set.
Figure 4 shows for each enhancement method the fraction of images with average score larger than a realism score and the overall AUC values. OHA results are often non-realistic, which is not surprising given their approach uses colors far from those in the original image. Our manipulated images are mostly realistic and similar to WSR and HAG in the level of realism. Recall, that this is achieved while our success in obtaining the desired enhancement effect is much better.
Controlling the Level of Enhancement: One of the advantages of our approach over previous ones is the control we provide the user over the degree of the manipulation effect. Our algorithm accepts a single parameter from the user, , which determines the level of enhancement. The higher is, the more salient will the region of interest become, since our algorithm minimizes , i.e., it aims to achieve . While we chose for most images, another user could prefer other values to get more or less prominent effects. Figure 5 illustrates the influence on the manipulation results.
The user-provided mask: In our dataset, the mask was marked by users to define a salient object in the scene. In order to use our method on a new image the user is required to mark the region that input region. Note that similarly to other imaging tasks, such as, image completion, compositing, recoloring and warping, the definition of the target region is up to the user to determine and is not part of the method. To facilitate the selection the user can utilize interactive methods such as [21, 28, 32] to easily generate region-of-interest masks.
Since our framework allows both increasing and decreasing of saliency it enables two additional applications: (i) Distractor Attenuation, where the target’s saliency is decreased, and (ii) Background Decluttering, where the target is unchanged while salient pixels in the background are demoted. A nice property of our approach is that all that is required for these is using a different mask setup, as illustrated in Figure 6.
Distractor Attenuation: The task of getting rid of distractors was recently defined by Fried et al. . Distractors are small localized regions that turned out salient against the photographer’s intentions. In  distractors were removed entirely from the image and the holes were filled by inpainting. This approach has two main limitations. First, it completely removes objects from the image thus changing the scene in an obtrusive manner that might not be desired by the user. Second, hole-filling methods hallucinate data and sometimes produce weird effects.
Instead, we propose to keep the distractors in the image while reducing their saliency. Figure 7 presents some of our results and comparisons to those obtained by inpainting. We succeed to attenuate the saliency of the distractors, without having to remove them from the image.
Background Decluttering: Reducing saliency is also useful for images of cluttered scenes where one’s gaze dynamically shifts across the image to spurious salient locations in the background. Some examples of this phenomena and how we attenuate it are presented in Figure 8. This scenario resembles that of removing distractors, with one main difference. Distractors are usually small localized objects, therefore, one could potentially use inpainting to remove them. Differently, when the background is cluttered, marking all the distractors could be tedious and removing them would result in a completely different image.
Our approach easily deals with cluttered background. The user is requested to loosely mark the foreground region. We then leave the foreground unchanged and manipulate only the background, using to automatically decrease the saliency of clutter pixels. The optimization modifies only background pixels with high saliency, since those with low saliency are represented in and therefore are matched to themselves.
We propose a general visual saliency retargeting framework that manipulates an image to achieve a saliency change, while providing the user control over the level of change. Our results outperform the state of the art in object enhancement, while maintaining realistic appearance. Our framework is also applicable to other image editing tasks such as distractors attenuation and background decluttering. Moreover, We establish a benchmark for measuring the effectiveness of algorithms for saliency manipulation.
Our method is not without limitations. First, since we rely on internal patch statistics, and do not augment the patch database with external images, the color transformations are limited to the color set of the image. Second, since our method is not provided with semantic information, in some cases the manipulated image may be non-realistic. For example, in Figure 7, the balloon is colored in gray, which is an unlikely color in that context. Despite its limitations, our technique often produces visually appealing results that adhere to the user’s wish.
We will release the data and code for our algorithm.
Visual saliency based on multiscale deep features.In CVPR, June 2015.
Automatic photo adjustment using deep neural networks.ACM TOG, 2015.