Single image super-resolution by approximated Heaviside functions

03/12/2015 ∙ by Liang-Jian Deng, et al. ∙ Case Western Reserve University, 0

Image super-resolution is a process to enhance image resolution. It is widely used in medical imaging, satellite imaging, target recognition, etc. In this paper, we conduct continuous modeling and assume that the unknown image intensity function is defined on a continuous domain and belongs to a space with a redundant basis. We propose a new iterative model for single image super-resolution based on an observation: an image is consisted of smooth components and non-smooth components, and we use two classes of approximated Heaviside functions (AHFs) to represent them respectively. Due to sparsity of the non-smooth components, a L_1 model is employed. In addition, we apply the proposed iterative model to image patches to reduce computation and storage. Comparisons with some existing competitive methods show the effectiveness of the proposed method.



There are no comments yet.


page 4

page 7

page 9

page 10

page 11

page 13

page 14

page 16

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Image super-resolution (SR) is to estimate a high-resolution (HR) image from one or multiple low-resolution (LR) images. When there is only

one low-resolution input, it is called single image super-resolution. The other case is called multiple image super-resolution. Compared to multiple image super-resolution, single image super-resolution is more practical when only one low-resolution image is available. Obviously, it is more challenging. In this paper, we address single image super-resolution. We can only collect low-resolution images sometimes because of the limitation of hardware devices and high cost. For instance, it needs more cost and time to get high-resolution images in medical imaging MRI. Due to long distance and air turbulence, we can not get high-resolution images for synthetic aperture radar (SAR) and satellite imaging. In target recognition, we may not always get high-resolution videos to make recognition accurately. Thus, it is important and useful to develop a more effective image super-resolution algorithm.

There are mainly four categories of image super-resolution methods: interpolation-based methods, learning-based methods, statistics-based methods and other methods. These methods are however not completely independent. For instance, some of statistics-based methods may involve learning strategies.

Nearest-neighbor interpolation and bicubic interpolation are two classical interpolation methods. Nearest-neighbor interpolation estimates intensity of an unknown location using that of its nearest neighbor point. It however often creates jaggy effect. Bicubic interpolation is to interpolate unknown intensity by utilizing a cubic kernel, and may cause blur effect. More interpolation methods have been proposed recently [34, 36, 53, 7, 22, 21, 46]. In [21], it introduces a contour stencils method to estimate the image contours based on total variation (TV) along curves. This method can distinguish lines of different orientations, curves, corners, etc., with a computationally efficient formula so that the resulted high-resolution images preserve image details well. In [46], Wang et al. propose a fast image super-resolution method on the displacement field. This method is based on two stages, one stage uses an interpolation technique to pick up the low-frequency image and the other employs a reconstruction method to recover local high-frequency structures.

Recently, many image super-resolution methods are learning-based [20, 19, 39, 42, 24, 18, 38, 31, 30, 35, 55, 48, 40, 9, 16, 51, 50, 49, 54, 26, 13, 52, 32, 3, 43, 44, 12]. Learning-based methods need two large training data sets, one formed by low-resolution images and the other by the corresponding high-resolution images. We learn a relation between the two training data sets, and then apply the relation to a given low-resolution image to obtain a high-resolution image. They rely heavily on the selection of training data and involves expensive computation. They are not single image super-resolution approaches in strict sense as they require extra training data. In [50], Yang et al. propose a sparse signal representation method for single image super-resolution. This method jointly trains two dictionaries for the low-resolution and high-resolution image patches, then applies the generated dictionaries to a low-resolution input to get the corresponding high-resolution output. In [12]

, Dong et al. first apply a deep learning method for single image super-resolution. The method learns a mapping between low-resolution and high-resolution images. The mapping is represented by a deep convolutional neural network (CNN) that takes the low-resolution image as the input and outputs the high-resolution image.

Statistics-based methods such as Maximum a Posterior (MAP) and Maximum Likelihood estimator (MLE) are a popular tool for image super-resolution [14, 6, 15]. Fattal [15] utilizes statistical edge dependency relating edge features in low and high resolution images to preserve sharp edges. In [14], an alternative approach using norm minimization and a bilateral prior based robust regularization, is proposed by Farsin et al.

Many other methods have also been proposed for image super-resolution. Examples are hybrid method [11], pixel classification method [1, 2], frequency technique [4], reconstruction method [27, 28, 33, 10, 41] and others [45, 5, 17, 37, 8].

Most existing image super-resolution methods are based on discrete models. In this paper, we propose a continuous model based on approximated Heaviside functions (AHFs) for single image super-resolution. Because an image normally consists of smooth components and non-smooth components, we model an image as a sum of two classes of AHFs, one representing smooth components and the other depicting non-smooth components, e.g., step edges. Due to the sparsity of non-smooth components, we design a regularization model and solve it by alternating direction method of multipliers (ADMM). An iterative model based on the model is proposed to preserve more details. We apply the coefficients, computed by the iterative model, to generate high-resolution images at any upscaling factors. In particular, the iterative model is applied to image patches, aiming to get cheap computation and storage. For images with smooth backgrounds, the proposed method may cause ring artifacts and we develop a strategy utilizing image gradient to reduce them. Numerical experiments show the proposed method is a competitive method comparing with existing excellent methods.

To the best of our knowledge, this is the first work to use AHFs on image super-resolution. The proposed method can preserve not only edges, but also the high-frequency details on non-edge regions. The proposed method is completely single image super-resolution method without using training data.

The organization of this paper is as follows. In Section 2, we review Heaviside function and show why we can use it for image super-resolution. In Section 3, we present the proposed method, including the corresponding new model, new algorithm and other strategies. Visual and quantitative experiments are given in Section 4. Finally, we draw conclusions in Section 5.

2 Heaviside function

Heaviside function, or Heaviside step function, is defined as follows (see Fig. 1(a))


which is singular at . In practice, we usually use its approximation, called approximated Heaviside function (AHF), such as and which approximate to when . In this paper, we use the following approximated Heaviside function,


where actually controls the smoothness. The smaller , the sharper edge (see Fig. 1(b)).

From [29], we know that any function in , , has a best approximation by linear combinations of characteristic functions of half-space, and is any positive integer. Let be a set of functions on defined as


where is an approximated Heaviside function. is the set of characteristic function of closed half-spaces of .

Theorem 1 (see [29]) For any positive integer , define as , where and and , then it is known that is dense in , .

Theorem 2 (see [29]) For every positive integers and every , is approximately a compact subset of .

In practical computing, one only can afford to use for a finite .

In our work, we focus on image super-resolution, i.e., . We assume the underlying image intensity function is defined on and with , can be approximated by the following equation,


where , and . We discreatize to denote different directions, and to denote discrete positions, , . We fix as the total number of pixels of the input image. Furthermore, with a specific is called a class of AHFs. In particular, we can describe edges of different orientations at the locations when setting a small . For an image , we assume it is a discretization of intensity function on , i.e., , , . We can rewrite Eq. (4

) as matrix-vector form,

, where with . Once we have computed coefficient , we can make image super-resolution with upscaling factor possible by the equation , where is obtained similarly with , .

Figure 1: (a) Heaviside function; (b) Two approximated Heaviside functions with (blue solid line) and (black dash line), the smaller the sharper edge (color images are better visualized in the pdf file).
Figure 2: Left panel: surface images of under and nine random parameter pairs ; Right panel: the corresponding 2D intensity images. From left to right and then from top to bottom: , , , , , , , , (for better visualization, we make slightly rotation to some surface images).
Figure 3: Left panel: surface images of under and the same nine parameter pairs as in Fig. 2; Right panel: the corresponding 2D intensity images. (for better visualization, we make slightly rotation to some surface images).

3 The proposed model and its solution

3.1 The proposed model and its solution

From Fig. 2 and Fig. 3, it is easy to know that the corresponding AHFs with big represent smooth components well, and the corresponding AHFs with small depict non-smooth components such as edges well. An image is generally consisted of smooth components and non-smooth components, thus we represent an image with two classes of AHFs, one with big parameter to represent smooth components (forming ) and the other with small to depict non-smooth components (forming ). By this strategy, the vector-form image , still denoted by , can be approximated by the following discrete formula,


where , and , are representation coefficients. The following task is to establish a model and compute the representation coefficients and , and then apply the computed coefficients to get super-resolution images by , where are generated similarly as using finer grids.

For an image, smooth components can be mainly represented by , non-smooth components such as edges are depicted by . Since non-smooth components are sparse in images. For instance, edges are not everywhere but distribute sparsely in most images. Thus we use sparsity on . The optimization model is


where , are regularization parameters, represents a low-resolution input.

Since term is not differentiable, we make a variable substitution for , and rewrite model (6) as


where is the substitution variable. In particular, Eq. (7) can be rewritten as


where , , and . The optimization problem (8) is separable w.r.t and . There are many methods to solve (8). We select alternating direction method of multipliers (ADMM)[23, 47, 25].

The augmented Lagrangian of Eq. (8) is


where is Lagrangian multiplier with proper size. The problem of minimizing is solved by iteratively and alternatively solving the following two subproblems:


We get the following ADMM algorithm for problem (8):

Algorithm 1
Input: Given low-resolution image , , , , ,
1. , , ,
2. while not converged do
4. solve subproblem (10) for , ,
5. solve subproblem (11) for , ,
6. .
7. End while.

The -subproblem (10) can be solved by least squares method:


where , , .

The -subproblem (11) has a closed form solution for each (see [47])


where and is assumed.

3.2 The iterative AHF method based on the proposed model

Even though model (6) takes different behaviors of smooth components and non-smooth components into consideration, we still see some blur in the high-resolution image. We find that the difference image , where is the resulted high-resolution output by model (6) and is a downsampling operator, contains some residual edges. To pick up more edges and details and thus to make results less blurry, we consider the difference as a new low-resolution input of model (6) to recompute a residual high-resolution image . This process is repeated until the residual edges are small enough. The sum of and its residual high-resolution images is the resulted super-resolution image. This iterative strategy can recover more image details (see Fig. 4). The following Algorithm 2 is the proposed iterative AHF algorithm for single image super-resolution. We use bicubic downsampling in our experiments although Algorithm 2 can work for general downsampling.

Algorithm 2 (Single image super-resolution via iterative AHF method)
Input: one low-resolution image (vector-form): , , , : upscaling factor.
: maximum number of iteration.
Output: high-resolution image (vector-form):
1. Construct matrices on coarse grids and on fine grids (see Section 3.1),
where .
2. Initialization: .
for k = 1:
a. Compute the coefficients: .
b. Update the high-resolution image: .
c. Downsampling to coarse grid: .
d. Compute residual: .
3. Assemble the high-resolution outputs: , .
4. Compute the final high-resolution image:
where conv represents a convolution operator, and is a Gaussian kernel with small size.

Note that step 2.a in Algorithm 2 is solved by Algorithm 1. From Fig. 5, we know that represents some edges. In particular, we use a Gaussian kernel with size and standard derivation 1 to make a convolution to reduce the oversharp information on the non-edge regions. In addition, although we introduce some parameters in the algorithm, they are not sensitive and easy to select. We will give a remark on parameter selection in Section 4. This is the proposed iterative method for single image super-resolution.

Figure 4: Illustration of step 2 of Algorithm 2 using “wing” image; (a) the sum image of (we set 3 iterations here) ; (b) the computed image for the first iteration. For better visualization, we add 0.3 to the intensities of and to brighten them. From the last two images, we can see that and pick up image details.
Figure 5: Illustration of step 4 of Algorithm 2 on pathes of a “wing image; (a) the sum image (see step 4 of Algorithm 2); (b) the image ; (c) the image which contains some edges and high-frequency information away from edges. For better visualization, we add 0.3 to the intensities of to brighten it.

3.3 Apply the proposed method to image patches

We have proposed the iterative AHF method for single image super-resolution, but it can be very expensive if we apply it to a whole image. For instance, if one low-resolution image has size and the upscaling factor is 2, then with 10 angles should have size . It is very expensive to implement Algorithm 2 with big non-sparse matrices. In our work, we apply the iterative AHF method to image patches, so that we can reduce computation and storage significantly. We set patch size to be and overlap to be 2 in our work. Moreover, pixels on the boundary will be handled by bicubic interpolation.

In summary, we use a set of Heaviside functions with sharp edges to represent sharp image edges which can be viewed as a kind of image high-frequency information. In particular, applying the proposed method to each image patches may make image high-frequency information more significant. This is the reason why the proposed method can pick up image high-frequency information.

4 Numerical experiments

In this section, two kinds of test images are utilized to illustrate the performance of different methods. One is low-resolution images without high-resolution ground-truth. The other is low-resolution images downsampled from known high-resolution images, for which we can make quantitative comparisons. All experiments are implemented in MATLAB(R2010a) on a laptop of 3.25Gb RAM and Intel(R) Core(TM) i3-2370M CPU: @2.40 GHz.

We compare our method with some competitive image super-resolution methods: nearest-neighbor interpolation, bicubic interpolation, iterative back-projection method (denoted as “BP” [28]), a edge-directed interpolation (denoted as “01’TIP” [34]), an edge-guided interpolation (denoted as “06’TIP” [53]), a kernel regression method (denoted as “07’TIP” [41]), a fast upsamling method (denoted as “08’TOG” [37]), TV-based super-resolution method (denoted as “11’JMIV” [8]), three state-of-the-art learning-based methods (denoted as “04’CVPR” [9], “10’TIP” [50] and “12’BMVC” [3]) and three state-of-the-art interpolation methods (denoted as “11’IPOL” [22], “11’SIAM” [21] and “14’TIP” [46]).

For gray images, we can apply the proposed Algorithm 2 to them directly. For color images such as RGB, we transform test images in RGB color space to “Ycbcr” color space which is very popular in image/video processing. “Y” represents luma component, “Cb” and “Cr” are blur-difference and red-difference components, respectively. We only apply the proposed algorithm to the illuminance channel because humans are more sensitive to luminance changes. In addition, we interpolate the color layers (Cb, Cr) using bicubic interpolation. After getting the upsampled images in “Ycbcr” color space, we transform them back to the original RGB color space for visualization.

We use root-mean-square error (RMSE) on illuminance channel to evaluate numerical performance,


where are the vector-form of ground-truth image and resulted super-resolution image, respectively. Note that although RMSE is a common numerical criterion, smaller RMSE can not always promise better visual results.

Remark on parameter selection: For the parameters in Algorithm 1 and Algorithm 2, we set , , , . Matrices with and with . We set 12 angles evenly distributed on , i.e., . We fix as the total number of pixels of the low-resolution patch, i.e., for patches. Note that, fine tuning of parameters for some images may get better results. However, we unify the parameter selection to illustrate the stability of the proposed method.

4.1 Super-resolution for generic images

From Fig. 6, we test low-resolution images without known high-resolution images. We compare the proposed method with nearest-neighbor interpolation, bicubic interpolation, two state-of-the-art learning-based methods (“10’TIP” [50] and “12’BMVC” [3]), a fast upsampling method (“08’TOG” [37]) and three state-of-the-art interpolation-based methods (“11’IPOL” [22], “11’SIAM” [21] and “14’TIP” [46]). The upscaling factors are all 3. From the resulted images, super-resolution images by bicubic interpolation generate blur effect. “08’TOG” performs well on image edges, but it smooths the details on the non-edge regions. The two learning-based methods (“10’TIP” and “12’BMVC”) perform comparable visually. In addition, “11’IPOL” and “11’SIAM” also show similar performance. In particular, “14’TIP” shows better property for image edges, but it smooths the image details on the non-edge regions. The proposed method outperforms the other methods, especially for image details on the non-edge regions.

In Fig. 7 and Fig. 8, low-resolution test images are generated by downsampling known high-resolution via bicubic interpolation. Because ground-truth images are known, we can evaluate quantitative results using RMSE. The upscaling factors are set to be 4 for “baboon” and “forest”, 3 for “field” and 2 for “leopard”. From these figures, we know that the results of “08’TOG” generate sharp edges but smooth out details on non-edge regions. TV-based method “11’JMIV” [8] also obtains sharp edges but has oil-painting effect. Results of iterative back-projection “BP” [28], the learning-based method (neighborhood embedding “04’CVPR” [9]) and the kernel regression method “07’TIP” [41] show significant blur. Because the proposed method can be viewed as an interpolation method, we compare our method with three state-of-the-art interpolation methods (“11’IPOL” [22], “11’SIAM” [21] and “14’TIP” [46]). In addition, we also compare the proposed method with two modern learning-based methods (“10’TIP” [50] and “12’BMVC” [3]). From Fig. 7 and Fig. 8, the proposed method obtains smaller RMSE than the three state-of-the-art interpolation methods and the two learning-based methods, and generates high-resolution images with more image details. Actually, learning-based methods need extra training data that can be viewed as extra prior information while our method does not require any extra data. Note that the interpolation-based method “14’TIP” [46] keeps very sharp image edges and runs quite fast, but it obtains bad RMSE and smooths image details significantly on the non-edge regions (see the close-ups). Furthermore, more quantitative results can be found in Tab. 1. It also demonstrates the effectiveness of the proposed method.

Figure 6: Results of “butterflywing” (first row) and “landscape” (second row), upscaling factors are all 3. From left to right: low-resolution images, results of bicubic interpolation, “10’TIP” [50], “08’TOG” [37], “11’IPOL” [22], “11’SIAM” [21], “12’BMVC” [3], “14’TIP” [46] and the proposed method (color images are better visualized in the pdf file).
Figure 7: Visual results and the corresponding RMSE of “baboon” and “forest” by upscaling factors of 4. Compared methods: bicubic interpolation, “BP” [28], “04’CVPR” [9], “08’TOG” [37], “11’JMIV” [8], “11’IPOL” [22], “11’SIAM” [21], “12’BMVC” [3], “14’TIP” [46], and the proposed method.
Figure 8: Visual results of “field” (upscaling factor 3) and “leopard” (upscaling factor 2) and the corresponding RMSE. Compared methods: nearest-neighbor interpolation, bicubic interpolation, “07’TIP” [41], “08’TOG” [37], “10’TIP” [50], “11’IPOL” [22], “11’SIAM” [21], “12’BMVC” [3], “14’TIP” [46], and the proposed method.

Bicubic 07’TIP [41] 08’TOG [37] 10’TIP [50] Ours
lena(2) 5.10 9.91 6.98 4.09 3.84
landscape(2) 10.75 13.55 11.74 10.09 9.63
purplebutterfly(2) 13.03 16.85 14.19 11.69 11.43
leaf(3) 17.51 18.23 17.55 17.39 17.28
dog(3) 2.12 3.76 1.95 1.93 1.85
fruits(3) 4.13 9.09 3.55 3.75 3.53
babyface(3) 5.12 8.44 4.86 4.87 4.74
train(4) 12.98 15.85 12.64 12.85 12.62
Table 1: RMSE of more test examples

4.2 Super-resolution for the images with smooth backgrounds

The proposed method performs well for natural images. It is a better method than the compared methods. However, the proposed method encounters a drawback: ring artifacts along the large scale edges of images which have smooth backgrounds (see Fig. 9(b)). The images with smooth backgrounds are generally obtained from single lens reflex (SLR) camera or other specific scenes. The ring artifacts mainly come from the added non-smooth components (see step 4 of Algorithm 2). In this section, we utilize a method without ring artifacts to make a mask, aiming to discard the ring artifacts of non-smooth components . We learn that bicubic interpolation will not cause ring artifacts (see the resulted images in section 4.1) and run very fast. Thus we use bicubic interpolation as our intermediate method to make the mask. Actually, we only need to update the in the step 4 of Algorithm 2 by which is obtained from the following equations, then we can reduce the ring artifacts significantly.




where is the vector norm of gradient at the location of image ; where “Bicubic” represents implementing bicubic interpolation to a low-resolution image and is a low-resolution image, is an upscaling factor; notation “” stands for dot product; is a thresholding value, and we set in our work. Obviously, this strategy reduces the ring artifacts of super-resolution images (see Fig. 9(c)).

In Fig. 10, we apply the proposed method to images “comic” and “face” which have smooth backgrounds. The upscaling factors are set to be 2 and 4, respectively. We can find more details from the close-ups in Fig. 10. From the figure, the proposed method reduces ring artifacts significantly and performs best, both visually and quantitatively.

In Fig. 11, we compare the proposed method with some competitive methods. The upscaling factor is 2. The test image “butterfly” has obvious smooth background. The results of bicubic interpolation and other two interpolation methods “01’TIP”, “06’TIP” show significant blur effect. Although the result by “08’TOG” and “14’TIP” keeps sharp edges well, they make image details of non-edge regions oversmoothing. The learning-based method “10’TIP” obtains the smallest RMSE and the competitive visual result. The proposed method preserves image details of non-edge regions better and overcomes ring artifacts significantly. In addition, RMSE of our method is almost the smallest one.

Figure 9: Results of “butterfly” by an upscaling factor of 2. Image “butterfly” is an image with the smooth background; (a) low-resolution image; (b) super-resolution image by the proposed method using original ; (c) super-resolution image by the proposed method using updated .
Figure 10: Results of “comic” (first row, upscaling factor 2) and “face” (second row, upscaling factor 4) and the corresponding RMSE. First row: ground-truth, bicubic interpolation, the learning-based method “10’TIP” [50] and the proposed method; Second row: ground-truth, the fast upsampling method “08’TOG” [37], TV-based method “11’JMIV” [8] and the proposed. Our method gets smallest RMSE and better visual results.
Figure 11: Results of “butterfly” by an upscaling factor of 2. First row: ground-truth, results of bicubic interpolation, an edge-directed interpolation “01’TIP” [34]; Second row: results of an edge-guided interpolation “06’TIP” [53], a fast upsampling method “08’TOG” [37], the learning-based method “10’TIP” [50]; Third row: results of two interpolation methods “11’IPOL” [22] and “14’TIP” [46], and the proposed method.

Parameters selection: The proposed method involves many parameters, e.g., , , , , patch size, etc. However, they are easy to select because the results are not sensitive to the selection of parameters (see , in Tab. 2 and , in Tab. 3). Actually, choosing suitable parameters is always a difficulty to many algorithms. Tuning empirically is a popular way for determining parameters. In our work, we obtain the parameters empirically. For instance, for and , we first fix one parameter , and then tune with 10 times change. For instance, we fix , then tune by , , , etc. When we find is the best one for , then tune in the rang of , e.g., and . Actually, the results around and also do not show obvious difference (see Tab. 3). Fine tuning of parameters might lead to slightly better results. But to make it simple, we set , in all test examples. Under this setting, the results are already good enough for comparisons. Similarly, we select the other parameters according to this way.

12.09 12.08 12.07
12.10 12.08 12.07
12.09 12.08 12.08
Table 2: RMSE around and
12.09 12.08 12.08
12.09 12.08 12.08
12.10 12.09 12.08
Table 3: RMSE around and

Computation time: From Fig. 12(a), we know that the computation time does not increase significantly when upscaling factor is increased from 2 to 9. Using our non-optimized Matlab code, it is below 10 seconds for a low-resolution image . From Fig. 12(b), when the size of low-resolution image is from to , the computation time is about increased from 3 seconds to 31 seconds. We can conclude that the computation time mainly depends on the size of low-resolution image but not so significantly on the scaling factor, since the main computation is to compute coefficients via the low-resolution image. Note that, there is a lot of room to speed up the speed. We can use cmex in Matlab to speed up the code that involves a lot of loops. We can also use parallel computing as the computation is done patch by patch.

Figure 12: (a) Computation time vs. upscaling factor for low-resolution image with size ; (b) Computation time vs. size of low-resolution image, the size of low-resolution image is increased from to and the upscaling factor is always set to be 5. The experimental computer is with 3.47GB RAM, Inter(R) Core(TM) i3-2130 CPU, @3.40GHz.

The relation between model (6) and model (6) combined with iterative idea: It is necessary to illustrate the relation between model (6) and model(6) combine with our iterative idea. Actually, there is only fine difference, especially in image details and edges, between model (6) and model (6) combined with iterative idea. Due to small magnitude of the difference, there is no significant difference between the two methods. In Fig. 13 we show an example. From Fig. 13, it is easy to know that the proposed model (6) combined with the iterative AHF method performs similarly RMSE with the proposed model (6). In addition, the error maps between the two strategies are almost same.

Figure 13: First column: results of model (6) with iterative AHF method; Second column: results of model (6) without iterative AHF method; Third column: error maps (for better vision, we add extra intensity to true error maps). Upscaling factors: 4.

5 Conclusions

In this paper, we casted image super-resolution problem as an intensity function estimation problem. We assumed the underlying intensity function, defined on a continuous domain and belonging to a space with redundant basis, could be approximately represented by two classes of approximated Heaviside functions (AHFs). The representation coefficients were computed by the proposed iterative AHF method using only one low-resolution image. We then applied the coefficients to get high-resolution images. To reduce computation and storage, we applied the proposed iterative AHF method to image patches. For images with smooth backgrounds, we designed a strategy utilizing image gradient to reduce ring artifacts along large scale edges. The method can be applied to any upscaling factors and needs no extra data for learning. In particular, we also discussed the parameter selection and computation time. Many experiments show that the proposed approach outperforms existing competitive methods, both visually and quantitatively.


The first and third authors thank the support by 973 Program (2013CB329404), NSFC (61370147, 61170311), Sichuan Province Sci. & Tech. Research Project (2012GZX0080). The first author is also supported by NSFC (61402082), Fundamental Research Funds for the Central Universities and Outstanding Doctoral Students Academic Support Program of UESTC.


  • [1] C. B. Atkins, C. A. Bouman, and J. P. Allebach. Tree-based resolution synthesis. International Conference on Image Processing (ICIP), pages 405–410, 1999.
  • [2] C. B. Atkins, C. A. Bouman, and J. P. Allebach. Optimal image scaling using pixel classification. International Conference on Image Processing (ICIP), pages 864–867, 2001.
  • [3] M. Bevilacqua, A. Roumy, C. Guillemot, and M.L.A. Morel. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. BMVC, 2012.
  • [4] S. Borman and R. L. Stevenson. Super-resolution from image sequences - a review. Midwest Symposium on Circuits and Systems, pages 374–378, 1998.
  • [5] E. Cands and C. Fernandez-Granda. Towards a mathematical theory of super-resolution. To appear in Communications on Pure and Applied Mathematics.
  • [6] D. Capel and A. Zisserman. Super-resolution enhancement of text image sequences.

    International Conference on Pattern Recognition (ICPR)

    , 1:600–605, 2000.
  • [7] Y. Cha, G. Lee, and S. Kim. Image zooming by curvature interpolation and iterative refinement. SIAM Journal on Imaging Sciences, 7:1284–1308, 2014.
  • [8] A. Chambolle and T. Pock. A first-order primal-dual algorithm for convex problems with applications to imaging. Journal of Mathematical Imaging and Vision, 40:120–145, 2011.
  • [9] H. Chang, D. Yeung, and Y. Xiong. Super-Resolution through neighbor embedding. Computer Vision and Pattern Recognition (CVPR), 1, 2004.
  • [10] P. Chatterjee, S. Mukherjee, S. Chaudhuri, and G. Seetharaman. Application of Papoulis-Gerchberg method in image super-resolution and inpainting. The Computer Journal, 52:80–89, 2007.
  • [11] G. Daniel, S. Bagon, and M. Irani. Super-Resolution from a single image. ICCV, pages 349–356, 2009.
  • [12] C. Dong, C. Loy, K. He, and X. Tang. Learning a deep convolutional network for image super-resolution. ECCV, 2014.
  • [13] W. Dong, G. Shi, L. Zhang, and X. Wu. Superresolution with nonlocal regularized sparse representation. Proceeding of SPIE, 2010.
  • [14] S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar. Fast and robust multiframe super resolution. IEEE transactions on image processing, 13:1327–1344, 2004.
  • [15] R. Fattal. Image upsampling via imposed edge statistics. ACM Transactions on Graphics, 26, 2007.
  • [16] C. Fernandez-Granda and E. Cands. Super-resolution via transform-invariant group-sparse regularization. ICCV, 2013.
  • [17] G. Freedman and R. Fattal. Image and video upscaling from local self-examples. ACM Trans. on Graphics (TOG), 30, 2011.
  • [18] W. T. Freeman, T. R. Jones, and E. C. Pasztor. Example-based super-resolution. IEEE Computer Graphics and Applications, 22:56–65, 2002.
  • [19] W. T. Freeman and E. C. Pasztor. Markov networks for super-resolution. Proceedings of 34th Annual Conference on Information Sciences and Systems, 2000.
  • [20] W. T. Freeman, E. C. Pasztor, and O. T. Carmichael. Learning low-level vision. International Journal of Computer Vision, 40:25–47, 2000.
  • [21] P. Getreuer. Contour stencils: Total Variation along curves for adaptive image interpolation. SIAM Journal on Imaging Sciences, 4:954–979, 2011.
  • [22] P. Getreuer. Image interpolation with contour stencils. Image Processing On Line, 1, 2011.
  • [23] T. Goldstein and S. Osher. The split bregman method for l1-regularized problems. SIAM Journal on Imaging Sciences, 2:323–343, 2009.
  • [24] E. Gur and Z. Zalevsky. Single-Image digital super-resolution a revised Gerchberg-Papoulis algorithm. IAENG International Journal of Computer Science, 34:251–255, 2007.
  • [25] B. He, M. Tao, and X. Yuan. Alternating direction method with gaussian back Substitution for separable Convex programming. SIAM Journal on Optimization, 22:313–340, 2012.
  • [26] L. He, H. Qi, and R. Zaretzki. Beta process joint dictionary learning for coupled feature spaces with application to single image super-resolution. CVPR, pages 345–352, 2013.
  • [27] M. Irani and S. Peleg. Super resolution from image sequence. Proceedings of 10th International Conference on Pattern Recognition (ICPR), pages 115–120, 1990.
  • [28] M. Irani and S. Peleg. Motion analysis for image enhancement: resolution, occlusion and transparency. Journal of Visual Communication and Image Representation, 4:324–335, 1993.
  • [29] P. C. Kainen, V. Krkov, and A. Vogt. Best approximation by linear combinations of characteristic functions of half-space. Journal of Approximation Theory, 122:151–159, 2003.
  • [30] C. Kim, K. Choi, K. Hwang, and J. B. Ra. Learning-based super-resolution using a multi-resolution wavelet approach. Iternational workshop on Advance Image Technology (IWAIT), 2009.
  • [31] C. Kim, K. Choi, and J. B. Ra. Improvement on learning-based super-resolution by adopting residual information and patch reliability. IEEE International Conference on Image Processing (ICIP), pages 1197–1200, 2009.
  • [32] K. I. Kim and Y. Kwon. Single-image super-resolution using sparse regression and natural image prior. IEEE Trans. Pattern Analysis and Machine Intelligence, 32:1127–1133, 2010.
  • [33] K. Komatsu, T. Igarashi, and T. Saito. Very high resolution imaging scheme with multiple different-aperture cameras. Signal Processing: Image Communication, 5:511–526, 1993.
  • [34] X. Li and M. Orchard. New edge-directed interpolation. IEEE Trans. Image Processing, 10:1521–1527, 2001.
  • [35] Liyakathunisa and V. K. Ananthashayana. Super resolution blind reconstruction of low resolution images using wavelets based fusion. International Journal of Computer and Information Engineering, 2:106–110, 2008.
  • [36] N. Mueller, Y. Lu, and M. Do. Image interpolation using multiscale geometric representations. SPIE proceedings, 2007.
  • [37] Q. Shan, Z. Li, J. Jia, and C. Tang. Fast image/video upsampling. ACM Transactions on Graphics (TOG), 27, 2008.
  • [38] J. Sun, J. Sun, Z. Xu, and H.-Y. Shum. Image super-resolution using gradient profile prior. CVPR, pages 1–8, 2008.
  • [39] J. Sun, N. N. Zheng, H. Tao, and H. Shum. Image hallucination with primal sketch priors. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2:729–736, 2003.
  • [40] Y.-W. Tai, S. Liu, M. Brown, and S. Lin. Super resolution using edge prior and single image detail synthesis. CVPR, pages 2400–2407, 2010.
  • [41] H. Takeda, S. Farsiu, and P. Milanfar. Kernel regression for image processing and reconstruction. IEEE Transactions on Image Processing, 16:349–366, 2007.
  • [42] M. F. Tappen, B. C. Russell, and W. T. Freeman. Exploiting the sparse derivative prior for super-resolution and image demosaicing. IEEE Workshop on Statistical and Computational Theories of Vision, 2003.
  • [43] R. Timofte, V. De Smet, and L. Van Gool. Anchored neighborhood regression for fast example-based super-resolution. ICCV, pages 1920–1927, 2013.
  • [44] R. Timofte, V. De Smet, and L. Van Gool. A+: adjusted anchored neighborhood regression for fast super-resolution. ACCV, 2014.
  • [45] F. Viola, A. W. Fitzgibbon, and R. Cipolla. A unifying resolution-independent formulation for early vision. CVPR, pages 494–501, 2012.
  • [46] L. Wang, H. Wu, and C. Pan. Fast image upsampling via the displacement field. IEEE Trans. Image Processing, 23:5123–5135, 2014.
  • [47] Y. Wang, J. Yang, W. Yin, and Y. Zhang. A new alternating minimization algorithm for total variation image reconstruction. SIAM Journal on Imaging Sciences, 1:248–272, 2008.
  • [48] Q. Xie, H. Chen, and H. Cao. Improved example-based single-image superresolution. International Congress on Image and Signal Processing (CISP), 3:1204–1207, 2010.
  • [49] J. Yang, Z. Wang, L. Zhe, and T. Huang. Coupled dictionary training for image super-resolution. IEEE transactions on image processing, 21:3467–3478, 2011.
  • [50] J. Yang, J. Wright, T. Huang, and Y. Ma. Image super-resolution via sparse representation. IEEE transactions on image processing, 19:2861–1873, 2010.
  • [51] J. Yang, J. Wright, Y. Ma, and T. Huang. Image super-resolution as sparse representation of raw image patches. IEEE Conference on Computer Vision and Pattern Recongnition (CVPR), pages 1–8, 2008.
  • [52] R. Zeyde, M. Elad, and M. Protter. On single image scale-up using sparse-representations. Curves and Surfaces, Lecture Notes in Computer Science, 6920:711–730, 2012.
  • [53] L. Zhang and X. Wu. An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans. Image Processing, 15:2226–2238, 2006.
  • [54] Y. Zhao, J. Yang, Q. Zhang, S. Lin, Y. Cheng, and Q. Pan. Hyperspectral imagery superresolution by sparse representation and spectral regularization. EURASIP Journal on Advances in Signal Processing, 2011.
  • [55] H. Zheng, A. Bouzerdoum, and S. L. Phung. Wavelet based nonlocal-means superresolution for video sequences. IEEE International Conference on Image Processing (ICIP), pages 2817–2820, 2010.