Blind deconvolution is a fundamental problem in low level vision, and is always drawing research attentions [22, 20, 15, 14, 21]. Given a blurry image , blind deconvolution aims to recover a clear version , in which it is crucial to first estimate blur kernel successfully. Formally, the degradation of image blur is modeled as
where and are with size , is with size , is the 2D convolution operator and is usually assumed as random Gaussian noises. Blind deconvolution needs to jointly estimate blur kernel and recover clear image .
The most successful blind deconvolution methods are based on the maximum-a-posterior (MAP) framework. MAP tries to jointly estimate and by maximizing the posterior , which can be further reformulated as an optimization on regularized least squares ,
where and are prior functions designed to prefer a sharp image and an ideal kernel, respectively. It is not trivial to solve the optimization problem in Eqn. (2), and instead it is usually addressed as alternate steps,
In the most blind deconvolution methods, kernel size is hyper-parameters that should be manually set. An ideal choice is the ground truth size to constrain the support domain, which however is not available in practical applications, requiring hand-crafted tuning.
On one hand, a smaller kernel size than ground truth cannot provide enough support domain for estimated blur kernel. Therefore, kernel size in the existing methods is usually pre-defined as a large value to guarantee support domain.
|truth size = 23||size = 23, err = 1.9||size = 47, err = 5.9||size = 69, err = 69.9|
On the other hand, as shown in Figure 1, oversized kernels are very likely to introduce estimation errors, and hence lead to unreasonable results. Hereby, we name this phenomenon larger-kernel effect. This interesting fact was first mentioned by Fergus . Then Cho and Lee  showed a similar result that the residual cost of (2) increases with over-estimated kernel size. However, such annoying phenomenon was not well analyzed and studied yet. Note that most MAP-based blind deconvolution algorithms adopt the trial-and-error strategy to tune kernel size, so the larger-kernel effect is a very common problem.
In this paper, we first explore the mechanism of larger-kernel effect and then propose a novel low rank-based regularization to relieve this adverse effect. Theoretically, we analyze the mechanism to introduce kernel estimation error in oversized kernel size. Specifically, we reformulate convolution of (3) and (4) to affine transformations and analyze their properties on kernel size. We show that for in sparse distributions, this larger-kernel effect remains with probability one. We also conduct simulation experiments to show that kernel error is expected to increase with kernel size even without noise . Furthermore, we attempt to find out a proper regularization to suppress noise in large kernels. By exploiting the low rank property of blur kernels, we propose a low-rank regularization to reduce noises in , suppressing larger-kernel effect. Experimental results on both synthetic and real blurry images validate the effectiveness of the proposed method, and show its robustness to against over-estimated kernel size. Our contributions are two-folds:
We give a thorough analysis to mechanism of the phenomenon that over-estimated kernel size yields inferior results in blind deconvolution, on which little research attention has been paid.
We propose a low rank-based regularization to effectively suppress larger-kernel effect along with efficient optimization algorithm, and performs favorably on oversized blur kernel than state-of-the-arts.
2 Larger-kernel effect
In this section, we describe the larger-kernel effect in detail and provide a mathematical explanation.
In Figure 1(b-c), it has shown that the larger the kernel size would lead to more inferior deblurring results, since the estimated blur kernel with larger support domain is very likely to introduce noises and estimation errors. Figure 1(a) shows both the error ratio (err)  of restored images and the Summed Squared Difference (SSD) of estimated kernels reach the lowest at the truth size and increase afterwards.
To analyze the source of larger-kernel effect, we firstly introduce an interesting fact that we call inflating effect.
For , we have . Hence, the Lebesgue measure of is zero, and the probability is zero. ∎
Claim 1 shows that padding linear independent columns to a thin matrix leads to a different least squares solution with lower residue squared cost.
The convolution part in (1
) is equivalent to linear transforms:
are required to be odd.
We attribute the larger-kernel effect to either substep (3) or (4). On one hand, remains identical when and increase by wrapping a layer of zeros around and the result of x-step keeps the same. Hence, x-step should not be blamed as the source of the larger-kernel effect. On the other hand, when is larger, will become inflated for the same . In 1D cases, where , assume , then
During blind deconvolution iterations, for identical values of , a larger introduces more columns onto both sizes of and results in different solutions. To illustrate this point, we tested a 1D version of blind deconvolution without kernel regularization and took different values of (truth and double and four times the truth size) for the 50th k-step optimization after 49 truth-size iterations (see Figure 2). Figure 2(a-c) show that the optimal solutions in different sizes differ slightly on the main body that lies within the ground truth size (colored in red), but greatly outside this range (colored in green) where zeros are expected. Figure 2(d-f) compare ground truth to estimated kernels in (a-c) after non-negativity and sum-to-one projections. Larger sizes yield more positive noises; hence, they lower the weight of the main body after projections and change the outlook of estimated kernel.
2.3 Probability of larger-kernel effect
Even if successfully iterates to truth , Claim 1 implicates the larger-kernel effect remains under the existence of random noise . We show
under which, the inflating effect holds for probability one in blind deconvolution.
Above all, we have
Kaltofen and Lobo  proved that for an M-by-M Toeplitz matrix composed of finite filed of elements,
Then we get the following claim:
See supplementary file. ∎
To now, we have shown that for in sparse distribution, the inflating effect happens almost surely.
2.4 Quantification of error increment
Assume iterates to ground truth during iterations. Then, for estimated kernel , we have
where represents Moore-Penrose pseudo-inverse. Then,
Assume , then
where and represents the smallest and the greatest singular values, respectively.
The inflating effect implicates that a larger kernel size amplifies the error in due to noise . To quantify this increment, we extracted a line from a clear image in Levin’s set  as shown in Figure 3(a), and plotted and with increasing kernel size . We also generated normalized random Gaussian vectors and compared to simulated boundaries of singular values (see Figure 3(b)). The error in increases hyper-linearly with kernel size.
In practice, nuances are expected between and . Cho and Lee  indicated that should be regarded as a sparse approximation to , not the ground truth. Hence,
which yields implicit noise . Assume , then,
To quantify how singular values of changes with kernel size, we simulated 100 times, in each of which we generated a stochastic sparse signal with length 254 under PDF in (10) with , and , and generated random Gaussian vector where . Figure 3(c) shows one example of generated and . Figure 3
(d) shows means and standard deviations of, and , which is the average of singular values, of simulated on . The error of is expected to grow with kernel size even .
3 Low-rank regularization
Blind deconvolution is an ill-posed problem for lacking sufficient information. Without regularization, MAP degrades to Maximum Likelihood (ML), which yields infinite solutions . As prior information, kernel regularization should be designed to compensate the shortage of ML and to guide the optimization to expected results. Great amount of studies focus on image regularization to describe natural images, , Total Variation (TV-) [16, 26, 23], hyper-Laplacian , dictionary sparsity [30, 12], patch-based low rank prior , non-local similarity  and deep discriminative prior .
Unfortunately, kernel optimization doesn’t attract much attention of the literature. Previous works adopted various kernel regularizations, e.g., -norm [28, 10, 3, 29, 21], -norm [15, 25, 20] and -norm , which, however, generally treated kernel regularization as an accessory and lacked a detailed discussion.
The larger-kernel effect is yielded by noise in ultra-sized kernels. Figure 1 and Figure 2 show that without kernel regularization, the main bodies of estimated kernels can emerge clearly, but increasing noises take greater amounts when is larger. To constrain to be clean, regularization is expected to distinguish noise from ideal kernels efficiently.
To suppress the noise in estimated kernels, we take low-rank regularization on such that k-step (4) becomes
Because the direct rank optimization is an NP-hard problem, continuous proximal functions are required. Fazel  proposed
as a heuristic proxy forwhere
is the N-by-N identity matrix andis a small positive number.
To allow this approximation to play a role in general matrices, the low-rank object is substituted to . The regularization function then becomes
where is the -th singular value of .
Taking low-rank regularization on kernels is motivated by a generic phenomenon of noise matrices . Figure 4(a-b) shows a non-negative Gaussian noise matrix and its singular values in decreasing order. For a noise matrix, where light and darkness alternate irregularly, the distribution of singular values decays sharply at lower indices; then, it breaks and drag a relatively long and flat tail to the last. In contrast, ideal kernels respond much lower to regularization (see Figure 4(c)). Based on this fact, noise matrices are distinguished by high cost from real kernels. Figure 4(d) shows that singular values of a low-rank regularized kernel are distributed similarly as the ground truth, compared with the impure one.
One intelligible explanation on the low-rank property of ideal kernels is the continuity of blur motions. Rank of a matrix equals the number of independent rows or columns; it reversely reflects how similar these rows or columns are. Speed of a camera motion is deemed to be continuous . Hence, the local trajectory of a blur kernel emerges similar to neighbor pixels, which is measured in a low value by the continuous proxy of rank.
Compared to previous norms, low-rank regularization responds more efficiently to noise. To illustrate this point, we generated a noisy kernel by adding a small percentage () of non-negative Gaussian noise and of the real kernel. Figure 5 shows that the low-rank cost rapidly adjust favorably to the noise but norms fail. That is because only takes statistical information. An extreme example consists of disrupting a truth kernel and randomly reorganizing its elements, with cost unchanged. In contrast, rank (singular values) corresponds to structural information.
Function is non-convex (and it is actually concave on ). To solve the low-rank regularized least squares (4), we introduce an auxiliary variable and reformulate the optimization into
Using the Lagrange method, (21) is solved by two alternate sub-optimizations
where is the iteration number while and are trade-off parameters.
The -substep is convex and accomplished using the Conjugate Gradient (CG) method. For -substep, low rank is adopted with limit; otherwise, the regularization may change the main body of kernel—an extreme result is . Thus, our strategy is to lower the rank at locally. Using the first-order Taylor expansion of at fixed matrix :
where is the
-th eigenvalue of, the k-substep in (22) is transformed into an iterative optimization
where is the inner iteration number. For convenience, we set as a flag (if , the k-substep will be skipped) and only tuned as the trade-off parameter.
Define the proximal mapping of function as follows:
Dong  proved that one solution to the proximal mapping of is
where is SVD of , and . Local low-rank optimization is implemented as iterations via the given parameter (see Algorithm 1). In our implementation, is designed to exponentially grow with to allow more freedom of for early iterations.
5 Experimental Results
In this section, we first discuss the effects of low rank-based regularization, then evaluate the proposed method on benchmark datasets, and finally demonstrate its effectiveness on real-world blurry images. The source code is available at https://github.com/lisiyaoATbnu/low_rank_kernel.
|size=23, err=1.55||size=47, err=1.56||size=69, err=2.14|
5.1 Effects of low rank-based regularization
Corresponding to high error ratios of large kernels in Figure 1, we repeat the experiment using same parameters except and . Figure 6 shows low-rank regularized kernels are much more robust to kernel size. Noises in kernels are efficiently reduced and qualities of restored images are enhanced. We further verify it on real-world images by imposing different regularization terms. As in Figure 7, blur kernels with low-rank regularization have less noises, while the others suffer from strong noises, yielding artifacts in the deblurring images. We note that in experiments of Figure 6 and Figure 7, we deliberately omitted multi-scaling scheme to expose the effectiveness of low-rank regularization itself.
5.2 Evaluation on synthetic dataset
The proposed method is quantitatively evaluated on dataset from . Figure 8 shows the success rates of state-of-the-art methods versus our implementations with and without (set and zero) low-rank regularization. The average PSNRs in Figure 8 with different sizes are compared in Table 1. Parameters are fixed during the whole experiment: , , , , and ; a 7-layer multi-scaling pyramid is taken. Kernel elements smaller than 1/20 of the maximum are cut to zero, which is also taken in [3, 14]. Low-rank regularization works more effectively than the regularization-free implementation and the state-of-art.
|Method||prior||truth size||double size|
|||Low rank (ours)||None (ours)||Low rank (ours)|
5.3 Evaluation on real-world blurry images
We compared our implementation to state-of-the-art methods on real-world images to reveal the robustness of low rank regularization on large kernel size. Specifically,  takes a heuristic iterative support domain detector based on the differences of elements of , which is regarded to be more effective than 1/20 threshold. Figure 9 shows that size yields strong noises in estimated kernels of previous works [3, 28], and even changes main bodies of kernels [15, 31]. In contrast, low rank regularization can keep the kernel relatively stable for the larger size. One more comparison of different regularizations and refinement methods on large kernel size are shown in Figure 10. As for computational efficiency of our method, it takes about 85s on a Lenovo ThinkCentre computer with Core i7 processor to process images with size .
In this paper, we demonstrate that over-estimated kernel sizes produce increased noises in estimated kernel. We attribute the larger-kernel effect to the inflating effect. To reduce this effect, we propose a low-rank based regularization on kernel, which could suppress noise while remaining restored main body of optimized kernel.
The success of blind deconvolution is contributed by many aspects. In practical implementations, even for noise-free , the intermediate is unlikely to iterate to ground truth, hence some parts of will be treated as implicit noises, which may intensify the effect even more than expected and require future researches.
This work is supported by the grants from the National Natural Science Foundation of China (61472043) and the National Key R&D program of China (2017YFC1502505). We thank Ping Guo for constructive conversation. Qian Yin is the corresponding author.
-  H. C. Andrews and B. R. Hunt. Digital image restoration, chapter 5.2, pages 102–103. Prentice-Hall, Englewood Cliffs, NJ, 1977.
-  T. F. Chan and C.-K. Wong. Total variation blind deconvolution. IEEE Trans. Image Process., 7(3):370–375, 1998.
-  S. Cho and S. Lee. Fast motion deblurring. ACM Trans. Graph., 28(5):145, 2009.
-  S. Cho and S. Lee. Convergence analysis of map based blur kernel estimation. arXiv preprint arXiv:1611.07752, 2016.
W. Dong, G. Shi, and X. Li.
Nonlocal image restoration with bilateral variance estimation: a low-rank approach.IEEE Transactions on Image Processing, 22(2):700–711, 2013.
-  W. Dong, G. Shi, X. Li, Y. Ma, and F. Huang. Compressive sensing via nonlocal low-rank regularization. IEEE Trans. Image Process., 23(8):3618–3632, 2014.
-  L. Fang, H. Liu, F. Wu, X. Sun, and H. Li. Separable kernel for image deblurring. In CVPR, pages 2885–2892. IEEE, 2014.
-  M. Fazel, H. Hindi, and S. P. Boyd. Log-det heuristic for matrix rank minimization with applications to hankel and euclidean distance matrices. In American Control Conf. (ACC), volume 3, pages 2156–2162, 2003.
-  R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T. Freeman. Removing camera shake from a single photograph. In ACM Trans. Graph., volume 25, pages 787–794, 2006.
-  D. Gong, M. Tan, Y. Zhang, A. Van den Hengel, and Q. Shi. Blind image deconvolution by automatic gradient activation. In CVPR, pages 1827–1836, 2016.
-  R. M. Gray. Toeplitz and circulant matrices: A review. Foundations and Trends in Communication and Information Theory, 2(3):155–239, 2006.
-  Z. Hu, J.-B. Huang, and M.-H. Yang. Single image deblurring with adaptive dictionary learning. In ICIP, pages 1169–1172. IEEE, 2010.
-  E. Kaltofen and A. Lobo. On rank properties of toeplitz matrices over finite fields. In Int. Symp. Symbolic and Algebraic Computation (ISSAC), pages 241–249, 1996.
-  D. Krishnan and R. Fergus. Fast image deconvolution using hyper-laplacian priors. In NIPS, pages 1033–1041, 2009.
-  D. Krishnan, T. Tay, and R. Fergus. Blind deconvolution using a normalized sparsity measure. In CVPR, pages 233–240, 2011.
-  A. Levin, R. Fergus, F. Durand, and W. T. Freeman. Image and depth from a conventional camera with a coded aperture. ACM Trans. Graph., 26(3):70, 2007.
-  A. Levin, Y. Weiss, F. Durand, and W. T. Freeman. Understanding and evaluating blind deconvolution algorithms. In CVPR, pages 1964–1971, 2009.
-  L. Li, J. Pan, W.-S. Lai, C. Gao, N. Sang, and M.-H. Yang. Learning a discriminative prior for blind image deblurring. In CVPR, pages 6616–6625. IEEE, 2018.
-  B. A. Olshausen and D. J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607, 1996.
J. Pan, Z. Lin, Z. Su, and M.-H. Yang.
Robust kernel estimation with outliers handling for image deblurring.In CVPR, pages 2800–2808, 2016.
-  J. Pan, D. Sun, H. Pfister, and M.-H. Yang. Blind image deblurring using dark channel prior. In CVPR, pages 1628–1636, 2016.
-  D. Perrone and P. Favaro. Total variation blind deconvolution: The devil is in the details. In CVPR, pages 2909–2916, 2014.
-  D. Ren, H. Zhang, D. Zhang, and W. Zuo. Fast total-variation based image restoration based on derivative augmented lagrangian method. Neurocomputing, 2015.
-  W. Ren, X. Cao, J. Pan, X. Guo, W. Zuo, and M.-H. Yang. Image deblurring via enhanced low-rank prior. IEEE Transactions on Image Processing, 25(7):3426–3437, 2016.
-  Q. Shan, J. Jia, and A. Agarwala. High-quality motion deblurring from a single image. ACM Trans. Graph., 27(3):73, 2008.
-  Y. Wang, J. Yang, W. Yin, and Y. Zhang. A new alternating minimization algorithm for total variation image reconstruction. SIAM Journal on Imaging Sciences, 1(3):248–272, 2008.
-  Y. Weiss and W. T. Freeman. What makes a good model of natural images? In CVPR, pages 1–8, 2007.
-  L. Xu and J. Jia. Two-phase kernel estimation for robust motion deblurring. In ECCV, pages 157–170, 2010.
-  L. Xu, S. Zheng, and J. Jia. Unnatural l0 sparse representation for natural image deblurring. In CVPR, pages 1107–1114, 2013.
-  X. Zhang, M. Burger, X. Bresson, and S. Osher. Bregmanized nonlocal regularization for deconvolution and sparse reconstruction. SIAM Journal on Imaging Sciences, 3(3):253–276, 2010.
-  W. Zuo, D. Ren, S. Gu, L. Lin, and L. Zhang. Discriminative learning of iteration-wise priors for blind deconvolution. In CVPR, pages 3232–3240, 2015.
Supplementary File: Proof to Theorem 2
Assume to be odd and (). Then,
For any -by- matrix A, i.f.f. . Thus,
As we know, the explicit formula of determinant of a Toeplitz matrix on its elements is unsolved in the current literature. Li  gives a concrete expression of by using LU factorization but fails to fit all situations ( when ). However, it can be shown that equals a multivariate polynomial function without manipulating the whole expression. By using Laplace expansion on , the item of largest degree is with factor 1.
Let be a continuous r.v. in the finite support domain [a, b]. Let be a polynomial function
where is a finite polynomial function with the largest degree less than . Generate a new r.v.
Then, for , the Cumulative Distribution Function (CDF)
, the Cumulative Distribution Function (CDF)is continuous at y.
Based on Beppo Levi’s Theorem,
Because ( is a constant), for , zeros of are finite, hence the Lebesgue measure of is zero. We have
Let be a continuous r.v. with PDF
For a sample of independent observations , generate a new r.v.
Based on the Law of Total Probability and Dominated Convergence Theorem,
is a polynomial function with the largest degree less than . Based on Lemma, we have
-  H. Li. On calculating the determinants of toeplitz matrices.J. Appl. Math. Bioinformatics, 1(1):55, 2011.