Image operators are fundamental building blocks for many computer vision tasks, such as image filtering and restoration. To obtain the desired results, many of these operators contain some parameters that need to be tweaked. We refer them as “parameterized image operators” in this paper. For example, parameters controlling the smoothness strength are widespread in most smoothing methods, and a parameter denoting the target upsampling scalar is always used in image super resolution.
Recently, many CNN based methods [1, 2, 3] have been proposed to approximate, accelerate or improve these parameterized image operators and achieved significant progress. However, we observe that the networks in these methods are often only trained for one specific parameter configuration, such as edge-preserving filtering  with a fixed smoothness strength, or super resolving low-quality images 
with a particular downsampling scale. Many different models need to be retrained for different parameter settings, which is both storage-consuming and time-consuming. It also prohibits these deep learning solutions from being applicable and extendable to a much broader corpus of images.
are highly unconstrained and probably very different for each. But can we find a common convolution weight space for different configurations by explicitly building their relationships? Namely, , where can be a linear or non-linear function. In this way, we can adaptively change the weights of the single target network based on in the runtime, thus enabling continuous parameter control.
To verify our hypothesis, we propose the first decoupled learning framework for parameterized image operators by decoupling the weights from the target network structure. Specifically, we employ a simple weight learning network as to directly learn the convolution weights of one task-oriented base network . These two networks can be trained end-to-end. During the runtime, the weight learning network will dynamically update the weights of the base network according to different input parameters, thus making the base network generate different objective results. This should be a very useful feature in scenarios where users want to adjust and select the most visually pleasant results interactively.
We demonstrate the effectiveness of the proposed framework for many different types of applications, such as edge-preserving image filtering with different degrees of smoothness, image super resolution with different scales of blurring, and image denoising with different magnitudes of noise. We also demonstrate the extensibility of our proposed framework on multiple input parameters for a specific application, and combination of multiple different image processing tasks. Experimental results demonstrate that the proposed framework is able to achieve almost as good results as the one solely trained for a single parameter value.
Despite of its generality and flexibility, all the convolution weights of need to be updated during the runtime. In the other words, the whole network needs to be re-evaluated when a new parameter is selected. This is very time-consuming and unacceptable in real user scenarios. To be adaptive for practical applications, the proposed framework can be further extended to dynamically change the weights of only one single layer while sharing most of the computation. With our default network structure, experiments demonstrate this cheap parameter-tuning version outperforms the state-of-the-art methods [4, 5] by a large margin.
As an extra bonus, the proposed framework makes it easy to analyze the underlying working principle of the trained task-oriented network by visualizing different parameters. The knowledge gained from this analysis may inspire more promising research in this area. To sum up, the contributions of this paper lie in the following four aspects.
We propose the first decoupled learning framework for parameterized image operators, where a weight learning network is learned to adaptively predict the weights for the task-oriented base network in the runtime.
We show that the proposed framework can be learned to incorporate many different parameterized image operators and achieve very competitive performance with the one trained for a single specific parameter or operator.
We extend our framework to enable cheap parameter tuning for real user scenarios, which outperforms many state-of-the-art methods by a large margin.
We provide a unique perspective to understand the working principle of the trained task-oriented network with some valuable analysis and discussion, which may inspire more promising research in this area.
2 Related Work
In the past decades, many different image operators have been proposed for low level vision tasks. Previous work [6, 7, 8, 9] proposed different priors to smooth images while preserving salient structures. Some papers [10, 11] utilized the spatial relationship and redundancy to remove unpleasant noise in the image. Other papers [12, 13, 14] aimed to recover a high-resolution image from a low-resolution image. Among them, many operators are allowed to tune some built-in parameters to obtain different results, which is the focus of this paper.
With the development of deep learning techniques, many different neural networks are proposed to approximate, accelerate and improve these operators[1, 3, 15, 16, 17]. But their common limitation is that one model can only handle one specific parameter setting. To enable all other parameters, enormous different models need to be retrained, which is both storage-consuming and time-consuming. By contrast, our proposed framework allows us to input continuous parameters to dynamically adjust the weights of the task-oriented base network. Moreover, it can even be applied to multiple different parameterized operators with one single network.
Recently, Chen et al.  conducted a naive extension for parameterized image operators by concatenating the parameters as extra input channels to the network. Compared to their method, where both the network structure and weights maintain the same for different parameters, the weights of our base network are adaptively changed. Experimentally we find our framework outperforms their strategy by integrating multiple image operators. By decoupling the network structure and weights, our proposed framework also makes it easier to analyze the underlying working principle of the trained task-oriented network, rather than leaving it as a black box as in many previous works like .
To enable practical image processing on mobile devices, a simple scheme to accelerate an operator is to apply it at a low-resolution image then upsample the result by reintroducing high-resolution details, which is used in bilateral upsampling  and the fast guided filter . To unify and generalize these two methods,  presents bilateral guided upsampling, which is further extended to  by incorporating such a technique into an end-to-end trained deep network. Though  and  are able to run very efficiently on even CPU devices, their resultant image quality is not good enough. By contrast, our cheap parameter-tuning extension directly runs the approximated operators on the original image resolution by sharing most of the computation costs for different operators, which not only enables real-time parameter adjustment on CPU devices but is also able to demonstrate its superior performance over these recent state-of-the-art approaches [4, 5] by a large margin.
Our method is also related to evolutionary computing and meta learning. Schmidhuber suggested the concept of fast weights in which one network can produce context-dependent weight changes for a second network. Some other works [22, 23, 24] casted the design of an optimization algorithm as a learning problem, Recently, Ha et al. 
proposed to use a static hypernetwork to generate weights for a convolutional neural network on MNIST and Cifar classification. They also leverage a dynamic hypernetwork to generate weights of recurrent networks for a variety of sequence modelling tasks. The purpose of their paper is to exploit weight sharing property across different convolution layers. But in our cases, we pay more attention to the common shared property among numerous input parameters and many different image operators.
3.1 Problem Definition and Motivation
The input color image and the target parameterized image operators are denoted as and respectively. transforms the content of locally or globally without changing its dimension. denotes the parameters which determine the transform degree of
and may be a single value or a multi-value vector. For example, insmoothing, is the balance weight controlling the smoothness strength, while in RTV filter 
, it includes one more spatial gaussian variance. In most cases,is a highly nonlinear process and solved by iterative optimization methods, which is very slow in runtime.
Our goal is to implement parameterized operator with a base convolution network . In previous methods like [3, 17], given a specific network structure of , separated networks are trained for different parameter configurations . In this way, the learned weights of these separated networks are highly unconstrained and probably very different. But intuitively, for one specific image operator, the weights of different might be related. So retraining separated models is too redundant. Motivated by this, we try to find a common weight space for different by adding a mapping constraint: , where can be a linear or non-linear function.
In this paper, we directly learn with another weight learning network rather than design it by handcraft. Assuming is a fully convolutional network containing a total of convolution layers, we denote their weights as respectively, then
where the input of is and the outputs are these weight matrices. In the training stage, and can be jointly trained. In the inference stage, given different input parameter , will adaptively change the weights of the target base network , thus enabling continuous parameter control.
Besides the original input image , the computed edge maps are shown to be a very important input signal for the target base network in . Therefore, we also pre-calculate the edge map of and concatenate it to the original image as an extra input channel:
where are the pixel coordinates and refers to the color channels.
To jointly train and , we simply use pixel-wise L2 loss in the RGB color space as  by default:
3.2 Network Structure
As shown in Fig. 1, our base network follows a similar network structure as . We employ 20 convolutional layers with the same kernel size, among which the intermediate 14 layers are formed as residual blocks. Except the last convolution layer, all the former convolutional layers are followed by an instance normalization 
layer and a ReLU layer. To enlarge the receptive field of
, the third convolution layer downsamples the dimension of feature maps by 1/2 using stride 2, and the third-to-last deconvolution layer (kernel size of) upsamples the downsampled feature maps to the original resolution symmetrically. In this way, the receptive field is effectively enlarged without losing too much image detail, and meanwhile the computation cost of intermediate layers is reduced. To further increase the receptive field, we also adopt the dilated convolution  as , more detailed network structure can be found in the supplementary material.
In this paper, the weight learning network simply consists of 20 fully connected (fc) layers by default. The fc layer is responsible to learn the weights for the convolutional layer, which can be written as following:
Where are the weight and bias of the fc layer. Assuming the parameter has a dimension of and has a dimension of . The dimension of and would be and respectively.
Note in this paper, we don’t intend to design an optimal network structure neither for the base network nor the weight learning network . On the contrary, we care more about whether it is feasible to learn the relationship between the weights of and different parameter configurations even by such a simple weight learning network .
3.3 Adaption to Cheap Parameter-Tuning
In the above default setting, the weights of all the convolution layers in are learned by the weight learning network and would be dynamically changed with the input parameters. That is to say, if users want to do some parameter tuning for different visual effects, the whole network needs to be re-evaluated. However in order to obtain the best image quality, most current methods like [1, 17, 18], including our default design, often requires to run through a complex deep networks and is very expensive for computational cost.
To tackle the efficiency issue, we further extend our framework from learning all the convolution weights to learning the weights of only one single layer. Since all the following layers behind this adjustable layer need to be re-evaluated when fed with varying input features, it is better to put this layer as deeper as possible in the base network. In this way, the computation of all the preceding layers can be shared by different parameters, and only the layers after this single adjustable layer need to be re-evaluated. This is of great practicability for many real scenarios.
Specifically, that is to say, only the weights of the th layer is the function of input parameter as Equation (4) while all the remaining weights are shared. During the runtime for parameter tuning, we only need to re-run the layers . Note that in our default network structure, can be either the weights of the convolutional layer or the scale and shift paramters in the following normalization layer. In this paper, by default, we choose the last instance normalization layer as the target layer and learn its scale and shift parameters, after which only one convolution layer needs to be run.
Though it seems more difficult for the network to adapt its behavior just with this single deep layer, we demonstrate its generality and effectiveness for different operators in the experiment section. Such a network design is able to outperform the state-of-the-art competitors [4, 5] by a large margin. More comprehensive ablation study about the learned layer type (convolution or instance normalization) and position is conducted in the analysis section.
4 Experiments on the proposed framework
To demonstrate the ability of our proposed framework in incorporating parameterised image operators while maintaining their accuracy, we evaluate the proposed decoupled learning framework with different training configurations as shown from subsection 4.3 to 4.4. We leverage two representative types of image processing tasks: image filtering and image restoration. Within each of them, more than four popular operators are selected for detailed experiments. Below, we briefly introduce all the operators and their implementation details as follows.
4.1 Choice of Image Operators
Image Filtering: here we employ six popular image filters, denoted as , WLS , RTV , RGF , WMF  and LLF , which have been developed specifically for many different applications, such as edge-preserving image smoothing, texture removal, detail exaggeration, image abstraction, and image enhancement.
smooth  - sharpening major image structures while eliminating a manageable degree of details by minimizing image gradients.
RTV  - extracting structures from textures by optimizing the new inherent variation and relative total variation measures.
WLS · - constructing edge-preserving multi-scale image decompositions for progressive coarsening of images.
RGF  - removing textures by controlling detail smoothing under a scale measure.
WMF  - an efficient 100+ times faster weighted median filter.
LLF  - fast local Laplacian filters for tone mapping.
Image Restoration: The goal of image restoration is to recover a clear image from a corrupted image. In this paper we deal with four representative tasks in this venue: super resolution , denoising , deblocking  and derain , which have been studied with deep learning based approaches extensively. Except for the derain task, all the others are tested on various parameter settings that indicate the corruption level of the input image.
- increasing the resolution or enhancing the lost details from a low-resolution blurry image, which is controlled by a downsampling scale with bicubic interpolation.
deblocking - recover the image details from a compressed JPEG image differentiated by the image quality factor.
derain - removing rain streaks from a captured rainy image.
4.2 Implementation Details
Dataset: We take use of the 17k natural images in the PASCAL VOC dataset as the clear images to synthesize the ground truth training samples. The PASCAL VOC images are picked from Flicker, and consists of a wide range of viewing conditions. To evaluate our performance, 100 images from the dataset are randomly picked as the test data for the image filtering task. While for the restoration tasks, we take the well-known benchmark for each specific task for testing, which is specifically BSD100 (super resolution), BSD68 (denoise), LIVE1 (deblock), RAIN12 (derain). For the filtering task, we filter the natural images with the aforementioned algorithms to produce ground truth labels. As for the image restoration tasks, the clear natural image is taken as the target image while the synthesized corrupted image is used as input.
Parameter Sampling: To make our network able to handle continuous parameters, we generate training image pairs with a much broader scope of parameter values rather than a single one. We uniformly sample parameters in either the logarithm or the linear space depending on the specific application. If the upper bound of the parameter range is tens or even hundreds of times larger than the lower bound, the parameters are sampled in the logarithm space to balance their magnitudes, otherwise they are sampled in the linear space.
4.3 Results on the Single Parameterized Operator
Image Filtering. We first evaluate the performance of six image operators with various controllable parameters individually. We train one network for each parameter value () in one operator, and also train a network jointly on continuous random values sampled from the operator’s parameter range, which can be inferred from the column in Table I. The performance of the two networks is evaluated on the test dataset with PSNR and SSIM error metrics. Since our goal is to measure the performance difference between these two strategies, we directly compute the absolute difference of their errors and demonstrate the results in Table I.
As can be seen from the table, though our proposed framework trained with numerous parameter settings lags a little behind the one trained on a single parameter value, their difference is very small especially for the visually more important error metric SSIM. For each image operator, previous methods usually requires to train separate networks for each parameter value, while our proposed approach only trains one single network jointly. Moreover, these image operators are dedicated to different image processing applications, the proposed framework is still able to learn all of them well, which verifies the versatility and robustness of our strategy.
Some visual results of our proposed framework are shown in Figure 2. As can be seen, our single network trained on continuous random parameter values is capable of predicting high-quality smooth images of various strengths.
Image Restoration. We then evaluate the proposed framework on three popular image restoration tasks as shown in Table II, which perform essentially different from image filtering. Our model learns to recover from different corrupted input images, instead of learning to obtain different visual effects given the same input image as in the image filtering task.
As shown in Table II, our results trained jointly on continuous random parameter values also show no big difference from the one trained solely on an individual parameter value, which further validate our algorithm in a broader image processing literature. Some corresponding visual results of our proposed framework are shown in Figure 3.
4.4 Results on Jointly Training Multiple Image Operators
Intuitively, another challenging case for our proposed framework is to incorporate multiple distinct image operators into a single learned neural network, which is much harder to be trained due to their different implementation details and purposes. To explore the potential of our proposed neural network, we experiment by jointly training over (i). 6 filtering based operators, (ii). 4 image restoration operators or (iii). all the 10 different operators altogether. To generate training images of each image operator, we sample random parameter values continuously within its parameter range. Since there is no parameter tuning for the derain task, we leverage its default parameter setting for training.
The input to the weight learning network now takes two parameters, one indicates the specific image operator while the other is the random parameter values assigned to the specified filter. These 10 image operators are denoted simply by 10 discrete values that range from 0.1 to 1.0 in the input parameter vector. Since the absolute parameter range may differ a lot from operator to operator, for example, [2,4] for super resolution and [0.002,0.2] for filter, we rescale the parameters in all the operators into the same numerical range to enable consistent back-propagated gradient magnitude.
As shown in Table III, training on each individual image operator achieves the highest numerical score (#ope.=1), which is averaged over multiple different parameter settings just like in previous tables. While jointly training over either 6 image filters or 4 restoration tasks (#ope.=6/4), even for the case where all 10 image operators are jointly trained (#ope.=10), their average performance degrades but still achieves close results to the best score. It means with the same network structure, our framework is able to incorporate all these different image operators together into a single network without losing much accuracy.
4.5 Comparison with Baseline 
We compare our proposed framework with one naive approach that employs only the base network with additional input channels as , which indicates the parameter values and image operators separately. Each additional channel is occupied with a single value.
The results are shown in Table III, which trains ten different image operators including both image filtering and restoration tasks together. We can see that the baseline from  lags behind us across all ten image operators. The potential reason for this phenomenon could be that it is more difficult to learn unified convolution weights to be suitable for tasks different in both goals and implementations. By contrast, the convolutional weights of our base network are adaptively changed for different tasks and input parameters. Theoretically speaking, our learned network should have the capability to express the base network with more numerous possibilities.
4.6 Generalization to Higher Dimensional Parameter Space
Except for experimenting on a single input parameter, we also demonstrate the experiments on inputting multiple types of parameters that belongs to either one image operator or various operators.
In this section, we evaluate the performance on the famous texture removal tool RTV . RTV has multiple parameters to adjust its visual effects, such as which balances between the data prior term and smoothness term in its energy function, which controls the spatial scale for computing the windowed variation and is even more effective in removing textures, that controls the sharpness of the final results and that refers to the number of iterations in the optimization.
We train our network on the continuous parameter range of first, therefore the input parameter is a one-element vector and constrained in the one dimensional space. Its results are shown in Table IV with ope. and dim. both equal to 1. Then we add one more parameter at a time to increase the input parameter dimension until all the aforementioned four parameters are included to form a four-element input vector, where dim. equals to 4. Moreover, following the experimental setting in Table III, we also train our network on 10 different image operators including the balance weight for RTV to form a two dimensional parameter space. We evaluate the performance on 5 discrete parameter values of that is commonly adopted for all the parameter dimensions.
We can see that when focusing on a single image operator (RTV) with up to even 4 parameters, the resultant performance of different parameter values are similar to each other in a reasonable range. But when we incorporate more image operators into joint training, the performance on RTV filter decreases significantly. A reasonable explanation for this phenomenon is that as the visual effects introduced by the additional input parameter are more different from the existing one, it becomes harder to incorporate the additional input parameter for joint training. Regarding the experiment in Table IV, the additional parameter for more image operators introduces more different visual effects compared to the four inherent parameters in the RTV filter, and hence becomes harder for joint training.
Note while incorporating more parameter dimensions into the training process, we don’t modify the amount of training samples and iterations. Each training sample is generated by randomly sampling the leveraged parameters.
5 Experiments on the cheap parameter-tuning version
In this section, we evaluate the cheap parameter-tuning adaption of our proposed framework with many recent filtering and restoration approaches. Most compared approaches design some complex and advanced deep neural network to achieve the optimal performance for their specific dedicated tasks. In order to balance between the performance and computation costs, we slightly modify the structure of by utilizing the depth-wise convolution  and increasing the intermediate feature channels (64 to 128) in this section. We validate and study the effectiveness of the proposed framework, which not only runs efficiently for parameter tuning but also achieves state-of-the-art performance compared to many recent deep learning approaches.
5.1 Image Filtering
We compare our proposed framework with two recent state-of-the-art approaches [4, 5], which share the closest spirit as ours in reproducing the image filters as well as possible while running in real-time by varying between different image operators.
To conduct on a broader scope of image operators like the other two approaches, besides the aforementioned six image filters, we further add four operators with more different visual effects for a fair comparison as follows.
LLF remapping  - fast local Laplacian filters for tone mapping by leveraging a remapping function.
WLS enhancement  - detail exaggeration implemented via multi-scale image decompositions.
Stylization  - transfer the “look” of one photographer’s masterpiece onto another photo via fast local Laplacian filters.
Abstraction  - pencil drawing by combining the tone and stroke structures, which complement each other in generating visually constrained results.
Note the six image filters mentioned in Section 4 contains controllable parameters that indicate different visual effects, while each of the above four operators represents a unique fixed visual effect. All these ten image operators are fed through the networks for joint training. The input parameters are sampled similarly as in section 4.4.
The image operators deployed in [4, 5] are relatively diverse, which include both photographic effects  and image structure manipulations  that are also demonstrated in our paper. Even if our deployed image operators are not exactly the same as theirs [4, 5], we believe the idea proposed in their paper is general enough, and we apply them to the same operators in our paper (some included and some not included in their papers).
In Table VI, we demonstrate the numerical results of our proposed framework compared with the competitors BGU  and DBL . For a fair comparison, the evaluation is conducted on one parameter setting (the default one) for each image operator following the other two approaches. As can be seen from the numerical errors (PSNR and SSIM), our results achieve significantly better results. The average PSNR over 10 image operators are about 5dB larger than the second best competitor.
Note both BGU and DBL learn to reproduce one specific operator with the default parameter setting, while our approach learns all the 10 image operators with their full range of parameter values jointly within one single network, which is much more difficult and challenging.
5.2 Image Restoration
Furthermore we compare our framework on the four aforementioned image restoration tasks: super resolution, denoising, deblocking and derain with many recent restoration methods as shown from Table VII to Table X. Except for the derain task, all the others are tested on various parameter settings that indicate the corruption level of the input image. Though these existing algorithms are dedicated on each specific field with specifically designed algorithms or network structures, our proposed framework is still able to achieve very competitive results on all these tasks. Need to note that all these tasks are jointly trained with one single network by our approach.
Given each new example for the restoration tasks, it needs to rerun the whole network from the beginning. Therefore it’s not a very valid case to justify our cheap parameter-tuning framework. However, theoretically this experiment still demonstrates the extraordinary ability of deep networks to incorporate different restoration capabilities within a very limited parameterized operation (the last instance normalization layer).
5.3 Running Time Comparison
In this section, we evaluate the running time of our proposed framework and the cheap parameter tuning module on different popular image resolutions. As shown in Table V, given a new input, we need to run through the whole model that takes less than 10 milliseconds for a 1080p image, but while switching between different image operations it takes only less than 1 millisecond on either GPU or CPU device, which is almost ten times faster than running the full model. Our implementation leverages the multi-core CPU package (NNPACK) in the MXNet framework for acceleration, and our model runs on a 20-core CPU device.
We also evaluate the running time of the baseline . Note their approach and ours share most of the base network structure, while their parameter tuning is implemented by adding additional input channels to the base network, which doesn’t increase the computation cost much. On the other hand, our framework includes one more weight learning network and hence lags behind them on the running time. From this table, we can also see that the weight learning network takes around 1.02 milliseconds to process a 1080p image, which is very computationally efficient.
|metric||method||WLS||RTV||RGF||WMF||LLF||LLF remap||WLS enhance||Stylization||Abstraction||Average|
|Quality||JPEG||ARCNN ||TNRD ||DnCNN ||MemNet ||Ours|
|Scale||Bicubic||SRCNN ||VDSR ||DRCN ||DnCNN ||MemNet ||Ours|
|BM3D ||WNNM ||TNRD ||DCDP ||Ours|
6 Understanding and analysis
To better understand the base network and the weight learning network , we analyze some meaningful behavior behind our framework in this section.
6.1 Complexity of the Model Size
The saved weights of our proposed decouple learning framework come from two parts, the weight learning network and base network. Let’s discuss our basic implementation where all the convolution weights are adaptively learned by the weight learning network with input parameters. The base network contains 2,432 instance normalization weights and 696,256 convolution weights to be learned. While the convolution weights are adaptively predicted by the weight learning network, and hence are not required to be saved. For each convolution layer in the base network, the weight learning network contains a single fully connected layer to predict its weights. Hence the learned weights for the fc layer are totally 2,088,768 (696,2563), where equals to 2 for the case of jointly training multiple image operators. Then the total weights required to save our decouple learning framework are 2,091,200 (2,088,768+2,432).
As analyzed above, most saved weights come from the weight learning network, which is equivalent to +1 times the convolution weights in the base network. Regardless of the detailed base network design, such a conclusion still holds. While jointly training multiple image operators, the total saved weights of our framework are almost 3 times the weights of the independent base
network, however it implements theoretically numerous image operations defined by the continuous input parameter range. In our PyTorch implementation, the saved model for jointly training 10 image operators with two input parameters only occupies 5.33 MB, which is sufficient for embedded systems.
While adapted for cheap parameter tuning, only the weights in a single instance normalization layer are learned by the weight learning network. The total saved weights of our framework are even closer to the one of an independent base network, therefore it becomes more efficient to jointly train multiple image operators. As the base network is bigger in this case, it takes 7.99 MB to save the whole model.
6.2 The Effective Receptive Field
In neuroscience, the receptive field is the particular region of the sensory space in which a stimulus will modify the firing of one specific neuron. The large receptive field is also known to be important for modern convolutional networks. Different strategies are proposed to increase the receptive field, such as deeper network structure or dilated convolution. Though the theoretical receptive field of one network may be very large, the real effective receptive field may vary with different learning targets. So how is the effective receptive field ofchanged with different parameters and ? Here we use smoothing  as the default example operator.
In Fig. 4, we study the effective receptive field of a non-edge point, a moderate edge point, and a strong edge point with different smoothing parameters respectively. To obtain the effective receptive field for a specific spatial point , we first feed the input image into the network to get the smoothing result, then propagate the gradients back to the input while masking out the gradient of all points except . Only the points whose gradient value is large than ( is the maximum gradient value of input gradient) are considered within the receptive field and marked as green in Fig. 4. From Fig. 4, we observe three important phenomena: 1) For a non-edge point, the larger the smoothing parameter is, the larger the effective field is, and most effective points fall within the object boundary. 2) For a moderate edge point, its receptive field stays small until a relatively large smoothing parameter is used. 3) For a strong edge point, the effective receptive field is always small for all the different smoothing parameters. It means, on one hand, the weight learning network can dynamically change the receptive field of based on different smoothing parameters. On the other hand, the base network itself can also adaptively change its receptive field for different spatial points.
6.3 Investigation into the Learned Weight
In this subsection, we discuss about the relationship between different learned convolutional kernels, and alternatives of the learned parameter position and type to better understand our proposed framework.
|metric||method||WLS||RTV||RGF||WMF||LLF||LLF remap||WLS enhance||Stylization||Abstraction||Average|
6.3.1 Difference of the learned convolution kernels between jointly trained network and solely trained network
To analyse the difference of the convolution weights between networks jointly trained on numerous random parameter values (“nume.”) and a single parameter value (“fixed”), we compute their correlation coefficient, individual mean and variance for each layer as shown in Table XI.
As can be seen, the correlation coefficient is almost 0 everywhere, which means there is no linear relationship between the two groups of convolution kernels. The absolute mean and variance of jointly trained network is also significantly smaller than that of the solely trained network. Therefore, in each way, their learned convolution weights are very different from each other, even if the learned smoothing effect is almost the same (PSNR/SSIM: 35.51dB/0.983 (nume.) vs. 35.83dB/0.982 (fixed)).
This simple experiment further verifies the huge solution space in the form of learned convolution kernels. Two exactly same results may be represented by very different convolution weights. The linear transformation in our proposed weight learning network actually connects all the different image operators and constrains their learned convolution weights in a limited high dimensional space.
6.3.2 Study of the learned parameter position and type
For most image operators presented in the paper, there is only one parameter needed. It might be too complex to use a weight learning network that only takes a scalar to manipulate all the convolution layers in the base network. To conduct a comprehensive study of the effects of where and what the learned controllable parameters are, we start by three experimental settings, (i). learned instance normalization parameters at different network depth. (ii). learned different types of convolution parameters at a fixed network depth. (iii). learned parameters in either all the convolutional or instance normalization layers.
(i). At first, we adjust the position of learned controllable parameters with four discrete network depth. It ranges from the first (1) instance normalization layer to the last (19) layer (the 20th convolutional layer is not followed by instance normalization). From the table, we can see that as the learned parameters become deeper in the network, the performance degrades. It’s a reasonable phenomenon, since the deeper the parameters are, the smaller the network capacity to differentiate various image operators is, and thus it’s harder to incorporate all the operators in the network.
We can also observe that the average error (PSNR) over all the 10 image operators only decreases by about 2.5dB from the first layer to the last layer, however it significantly improves the running speed when varying between different image operators in the network.
(ii). To explore the influence of other types of parameters in the network to separate these image operators, we replace the controllable parameters in the instance normalization layer with the ones in the convolutional layer.
Each convolutional layer contains a
weight tensor and a
-length bias vector.is the number of output feature channels, is the number of input feature channels, is the kernel size. equals to for the intermediate convolutional layers in our baseline network. Since the convolutional layer is followed by an instance normalization layer, the added bias in the convolutional layer will be balanced out by the following normalization operation and becomes useless. Therefore we experiment only by varying the parameters in the convolutional weights.
|single||single (base)||nume.||nume. (fc2R)||nume. (fc2)|
We study with three types of learned convolutional parameters, which are separately: “conv”, all the weight kernels (); “conv channel1”, one slice of weight kernels (); “conv channel2”, another slice of weight kernels (). The learned convolutional layer shares the same network depth as the last instance normalization layer.
As can be seen from the table, learning all the weight kernels in the convolutional layer (conv) achieves the highest performance, which is very reasonable. While learning all the convolutional kernels that are only related to one single input channel (conv channel1), the performance degrades by about 0.8dB. Learning all the parameters that are related to one single output channel (conv channel2) obtains the worst performance. Note even if “conv channel1” and “conv channel2” share the same number of controllable parameters, all the learned parameters in “conv channel2” only result in adjustment of a single channel of output feature map, which influence the following layers much less than “conv channel1” and hence achieve worse results.
Interestingly, by learning the convolutional parameters (conv and conv channel1) achieves better results than learning instance normalization parameters at the same network depth. This may be because the convolutional layer is ahead of the instance normalization, and the learned convolutional parameters (even for “conv channel1”, equals to 3 in our case) is much more than the normalization’s ().
(iii). To further explore the difference between learning convolution and normalization layers, we try to learn the parameters in either all the convolutional layers or all the instance normalization layers in the network, which should theoretically achieves the best performance of incorporating all the image operators.
As shown in the “conv(all)” and “norm(all)” rows, surprisingly they achieve almost the same performance on the either PSNR or SSIM error metric. This means it already touches the top performance limit of incorporating the 10 image operators with this neural network.
Note learning all the normalization parameters (norm(all)) only obtains a little improvement (0.41dB) than learning the one in the first layer (norm(1)), from which we can see it’s more important which position the controllable parameters are instead of how many layers the parameters are located in.
6.4 Interpretation of the Weight Learning Network
To help understand the connection between the base network and the weight learning network , we decompose the parameter vector and the weight matrix into independent elements and respectively, then:
where denotes convolution operation, and is the dimension of . In other words, the one convolution layer, whose weights are learned with one single fc layer, is exactly equivalent to a multi-path convolution block as shown in Figure 5. Learning the weight and bias of the single fc layer is equivalent to learning the common basic convolution kernels in the convolution block.
6.5 Analysis of More Variants of Our Proposed Network
In this subsection, we experiment with a few variants of our network to justify its effectiveness.
6.5.1 Training the base network only
Since training a fully convolutional network alone has been employed frequently by previous image processing papers [17, 1, 18], which presents a strong baseline, we experiment with this alternative (“nume. (base) ”) which only leverages the base network by training it on one specific parameter configuration. As show in Table XIII, our proposed framework (“single”) achieves even better performance than the baseline under the PSNR error metric.
6.5.2 Training with more fc layers
We also try a deeper weight learning network with more fully connected layers. Here we simply add one more fully connected layer to the default weight learning network, and demonstrate the performance of its two variants with (“nume. (fc2R)”) and without (“nume. (fc2)”) ReLU between these two layers respectively. As shown in Table XIII, they all achieve comparable performance with that of single layer (“nume.”) used in our paper. The potential reason for this phenomenon is that this one-layer weight learning network is sufficient for adaptively learning various parameter settings, while adding more weights/complexity to the network does not contribute to the performance much.
6.6 Interpolation Ability of the Proposed Framework on Unseen Input Parameters
Since the weight learning network contains a single fully connected layer with no non-linear activation layers, the predicted convolution weights should be a linear transformation of the input parameters. Given such a fact, any convolutional kernels of a specific parameter should be the linear interpolation of the other two. Hence we are curious about the interpolation ability of the proposed framework.
To verify such a property, we train the network using only a few fixed parameter values, which corresponds to the 5 non-boldface parameter values in Table XIV. But in the test stage, we use another 4 parameter values (boldface in Table XIV) that have not been seen by the network in the training stage but are between the lower and upper bound of its parameter range. As shown in Table XIV, we can see that the network performs similarly to the one trained with only one parameter value, but more importantly for the interpolated boldface parameter values that the network does not recognize, it also surprisingly achieves very comparable results.
This means that a few parameter values are already sufficient for learning a good linear transformation in the weight learning network from input parameters to convolution weights. However, as in a real scenario, the number of such fixed training parameters is usually difficult to decide, due to many different parameter ranges of image operators. As a result, we choose to sample random parameter values instead of only a few of them for training in our paper.
In this paper, we propose the first decouple learning framework for parameterized image operators, where the weights of the task-oriented base network are decoupled from the network structure and directly learned by another weight learning network . These two networks can be easily end-to-end trained, and dynamically adjusts the weights of for different parameters during the runtime. We show that the proposed framework can be applied to different parameterized image operators, such as image smoothing, denoising and super resolution, while obtaining comparable performance as the network trained for one specific parameter configuration. It also has the potential to jointly learn multiple different parameterized image operators within one single network. For real user scenarios, we further extend our framework to enable cheap parameter tuning, which obtains superior performance over state-of-the-art methods by a large margin. To better understand the working principle, we also provide some valuable analysis and discussions, which may inspire more promising research in this direction. More theoretical analysis is worthy of further exploration in the future.
-  Q. Fan, J. Yang, G. Hua, B. Chen, and D. Wipf, “A generic deep architecture for single image reflection removal and image smoothing,” in Proceedings of the 16th International Conference on Computer Vision (ICCV), 2017, pp. 3238–3247.
J. Kim, J. Kwon Lee, and K. Mu Lee, “Accurate image super-resolution using
very deep convolutional networks,” in
Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1646–1654.
L. Xu, J. Ren, Q. Yan, R. Liao, and J. Jia, “Deep edge-aware filters,” in
International Conference on Machine Learning, 2015, pp. 1669–1678.
-  J. Chen, A. Adams, N. Wadhwa, and S. W. Hasinoff, “Bilateral guided upsampling,” ACM Transactions on Graphics (TOG), vol. 35, no. 6, p. 203, 2016.
-  M. Gharbi, J. Chen, J. T. Barron, S. W. Hasinoff, and F. Durand, “Deep bilateral learning for real-time image enhancement,” ACM Transactions on Graphics (TOG), vol. 36, no. 4, p. 118, 2017.
-  L. Xu, C. Lu, Y. Xu, and J. Jia, “Image smoothing via l 0 gradient minimization,” in ACM Transactions on Graphics (TOG), vol. 30, no. 6. ACM, 2011, p. 174.
-  L. Karacan, E. Erdem, and A. Erdem, “Structure-preserving image smoothing via region covariances,” ACM Transactions on Graphics (TOG), vol. 32, no. 6, p. 176, 2013.
-  L. Xu, Q. Yan, Y. Xia, and J. Jia, “Structure extraction from texture via relative total variation,” ACM Transactions on Graphics (TOG), vol. 31, no. 6, p. 139, 2012.
-  Q. Zhang, X. Shen, L. Xu, and J. Jia, “Rolling guidance filter,” in European conference on computer vision. Springer, 2014, pp. 815–830.
-  A. Buades, B. Coll, and J.-M. Morel, “A non-local algorithm for image denoising,” in Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, vol. 2. IEEE, 2005, pp. 60–65.
-  M. Elad and M. Aharon, “Image denoising via sparse and redundant representations over learned dictionaries,” IEEE Transactions on Image processing, vol. 15, no. 12, pp. 3736–3745, 2006.
-  J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE transactions on image processing, vol. 19, no. 11, pp. 2861–2873, 2010.
-  J. Sun, Z. Xu, and H.-Y. Shum, “Image super-resolution using gradient profile prior,” in Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on. IEEE, 2008, pp. 1–8.
-  M. E. Tipping and C. M. Bishop, “Bayesian image super-resolution,” in Advances in neural information processing systems, 2003, pp. 1303–1310.
-  C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE transactions on pattern analysis and machine intelligence, vol. 38, no. 2, pp. 295–307, 2016.
-  V. Jain and S. Seung, “Natural image denoising with convolutional networks,” in Advances in Neural Information Processing Systems, 2009, pp. 769–776.
-  S. Liu, J. Pan, and M.-H. Yang, “Learning recursive filters for low-level vision via a hybrid neural network,” in European Conference on Computer Vision. Springer, 2016, pp. 560–576.
-  Q. Chen, J. Xu, and V. Koltun, “Fast image processing with fully-convolutional networks,” in IEEE International Conference on Computer Vision, vol. 9, 2017.
-  J. Kopf, M. F. Cohen, D. Lischinski, and M. Uyttendaele, “Joint bilateral upsampling,” ACM Transactions on Graphics (ToG), vol. 26, no. 3, p. 96, 2007.
-  K. He and J. Sun, “Fast guided filter,” arXiv preprint arXiv:1505.0099.
-  J. Schmidhuber, “Learning to control fast-weight memories: An alternative to dynamic recurrent networks,” Neural Computation, vol. 4, no. 1, pp. 131–139, 1992.
-  M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. De Freitas, “Learning to learn by gradient descent by gradient descent,” in Advances in Neural Information Processing Systems, 2016, pp. 3981–3989.
-  O. Wichrowska, N. Maheswaranathan, M. W. Hoffman, S. G. Colmenarejo, M. Denil, N. de Freitas, and J. Sohl-Dickstein, “Learned optimizers that scale and generalize,” in International Conference on Machine Learning, 2017.
-  Y. Chen, M. W. Hoffman, S. G. Colmenarejo, M. Denil, T. P. Lillicrap, and N. de Freitas, “Learning to learn for global optimization of black box functions,” in International Conference on Machine Learning, 2017, pp. 748–756.
-  D. Ha, A. Dai, and Q. V. Le, “Hypernetworks,” ICLR, 2017.
-  L. Xu, C. Lu, Y. Xu, and J. Jia, “Image smoothing via l0 gradient minimization,” ACM Transactions on Graphics (SIGGRAPH Asia), 2011.
-  D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis,” in Proc. CVPR, 2017.
-  F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” in ICLR, 2016.
-  Z. Farbman, R. Fattal, D. Lischinski, and R. Szeliski, “Edge-preserving decompositions for multi-scale tone and detail manipulation,” in ACM Transactions on Graphics (TOG), vol. 27, no. 3. ACM, 2008, p. 67.
-  Q. Zhang, L. Xu, and J. Jia, “100+ times faster weighted median filter (wmf),” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2830–2837.
-  M. Aubry, S. Paris, S. W. Hasinoff, J. Kautz, and F. Durand, “Fast local laplacian filters: Theory and applications,” ACM Transactions on Graphics (TOG), vol. 33, no. 5, p. 167, 2014.
-  C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a deep convolutional network for image super-resolution,” in European conference on computer vision. Springer, 2014, pp. 184–199.
-  X. Mao, C. Shen, and Y.-B. Yang, “Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections,” in Advances in neural information processing systems, 2016, pp. 2802–2810.
-  C. Dong, Y. Deng, C. Change Loy, and X. Tang, “Compression artifacts reduction by a deep convolutional network,” in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 576–584.
-  X. Fu, J. Huang, D. Zeng, Y. Huang, X. Ding, and J. Paisley, “Removing rain from single images via a deep detail network,” in IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1715–1723.
-  F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” CVPR, 2017.
-  C. Lu, L. Xu, and J. Jia, “Combining sketch and tone for pencil drawing production,” in Proceedings of the Symposium on Non-Photorealistic Animation and Rendering. Eurographics Association, 2012, pp. 65–73.
-  Y. Chen and T. Pock, “Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration,” IEEE transactions on pattern analysis and machine intelligence, vol. 39, no. 6, pp. 1256–1272, 2017.
-  K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising,” IEEE Transactions on Image Processing, vol. 26, no. 7, pp. 3142–3155, 2017.
-  Y. Tai, J. Yang, X. Liu, and C. Xu, “Memnet: A persistent memory network for image restoration,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 4539–4547.
-  J. Kim, J. K. Lee, and K. M. Lee, “Accurate image super-resolution using very deep convolutional networks,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR Oral), June 2016.
-  J. Kim, J. Kwon Lee, and K. Mu Lee, “Deeply-recursive convolutional network for image super-resolution,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1637–1645.
-  K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-d transform-domain collaborative filtering,” IEEE Transactions on image processing, vol. 16, no. 8, pp. 2080–2095, 2007.
-  S. Gu, L. Zhang, W. Zuo, and X. Feng, “Weighted nuclear norm minimization with application to image denoising,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2862–2869.
-  K. Zhang, W. Zuo, S. Gu, and L. Zhang, “Learning deep cnn denoiser prior for image restoration,” in IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, 2017.
-  X. Fu, J. Huang, X. Ding, Y. Liao, and J. Paisley, “Clearing the skies: A deep network architecture for single-image rain removal,” IEEE Transactions on Image Processing, vol. 26, no. 6, pp. 2944–2956, 2017.