# Weighted Nonlocal Total Variation in Image Processing

In this paper, a novel weighted nonlocal total variation (WNTV) method is proposed. Compared to the classical nonlocal total variation methods, our method modifies the energy functional to introduce a weight to balance between the labeled sets and unlabeled sets. With extensive numerical examples in semi-supervised clustering, image inpainting and image colorization, we demonstrate that WNTV provides an effective and efficient method in many image processing and machine learning problems.

## Authors

• 2 publications
• 13 publications
• 6 publications
06/05/2013

### Multiclass Total Variation Clustering

Ideas from the image processing literature have recently motivated a new...
02/26/2015

### Total variation on a tree

We consider the problem of minimizing the continuous valued total variat...
06/18/2012

### Total Variation and Euler's Elastica for Supervised Learning

In recent years, total variation (TV) and Euler's elastica (EE) have bee...
04/30/2012

### Elimination of Glass Artifacts and Object Segmentation

Many images nowadays are captured from behind the glasses and may have c...
04/19/2015

### Visual Recognition Using Directional Distribution Distance

In computer vision, an entity such as an image or video is often represe...
01/24/2022

### The Image Deblurring Problem: Matrices, Wavelets, and Multilevel Methods

The image deblurring problem consists of reconstructing images from blur...
01/25/2017

### Photographic dataset: playing cards

This is a photographic dataset collected for testing image processing al...
##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

Interpolation on point cloud in high dimensional space is a fundamental problem in many machine learning and image processing applications. It can be formulated as follows. Let be a set of points in and be a subset of . Let be a function on the point set and the value of on is given as a function over . The goal of the interpolation is to find the function on with the given values on .

Since the point set is unstructured in high dimensional space, traditional interpolation methods do not apply. In recent years, manifold learning has been demonstrated to be effective and attract more and more attentions. One basic assumption in manifold learning is that the point cloud samples a low dimensional smooth manifold, , embedded in . Another assumption is that the interpolation function is a smooth function in . Based on these two assumptions, one popular approach is to solve by minimizing the norm of its gradient in . This gives us an optimization problem to solve:

 minu∥∇Mu∥2,subject to:u(x)=g(x),x∈S, (1)

with

 ∥∇Mu∥2=(∫M|∇Mu(x)|2dx)1/2.

Usually, is approximated by nonlocal gradient

 Dyu(x)=√w(x,y)(u(y)−u(x)). (2)

Then, the discrete version of (1) is

 min∑x,y∈Pw(x,y)(u(x)−u(y))2,subject to:u(x)=g(x),x∈S, (3)

from which, we can derive a linear system to solve on point cloud . This is just the well known nonlocal Laplacian which is widely used in nonlocal methods of image processing BCM05 ; BCM06 ; GO07 ; GO08 . It is also called graph Laplacian in graph and machine learning literature  Chung:1997 ; ZhuGL03 . Recently, it was found that, when the sample rate is low, i.e. , graph Laplacian method fails to give a smooth interpolation Shi:harmonic ; Shi2017 . Continuous interpolation can be obtained by using point integral method Shi:harmonic or weighted nonlocal Laplacian Shi2017 .

In many problems, such as data classification or image segmentation, minimizing the total variation seems to be a better way to compute the interpolation function, since it prefers piecewise constant function in total variation minimization. This observation motives another optimization problem:

 min∥u∥TVM,subject to:u(x)=g(x),x∈S, (4)

with

 ∥u∥TVM=∫M|∇Mu(x)|dx.

Total variation model has been studied extensively in image processing since it was first proposed by Rudin, Osher and Fatemi(ROF) in rudin1992nonlinear . It is well known that total variation has the advantage of preserving edges, which is always preferable because edges are significant features in the image, and usually indicate boudaries of objects. Despite its good performance of restoring ”cartoon” part of the image, TV based methods fail to achieve satisfactory results when texture, or repetative structures, are present in the image. To address this problem, Buades et al proposed a nonlocal means method based on patch distances for image denoising BCM05 . Later, Gilboa and Osher GO07 ; GO08 formalized a systematic framework, include nonlocal total variation model, for nonlocal image processing.

Using nonlocal gradient to approximate the total variation, we can write down the discrete version of (4),

 min∑x∈P(∑y∈Pw(x,y)(u(x)−u(y))2)1/2,subject to:u(x)=g(x),x∈S. (5)

This problem can be solved efficiently by split Bregman iteration osher2005iterative ; doi:10.1137/080725891 . However, it was reported that LDMM , when the sample rate is low, above nonlocal TV model has the same defect as that in graph Laplacian approach (3). The interpolation obtained by solving above optimization problem is not continuous on the sample points.

In this paper, inspired by weighted nonlocal laplacian method proposed in Shi2017 , we propose a weighted nonlocal TV method (WNTV) to fix this discontinuous issue. The idea is to introduce a weight related to the sample rate to balance the labeled terms and unlabeled terms. More specifically, we modify model (5) a little bit by introducing a weight,

 minu∑x∈V∖S(∑y∈Vω(x,y)(u(x)−u(y))2)1/2+|V||S|∑x∈S(∑y∈Vω(x,y)(u(x)−u(y))2)1/2,

This optimization problem also can be solved by split Bregman iteration. Based on our experience, the convergence is even faster than the split Bregman iteration in the original nonlocal total variation model (5

). Using extensive examples in image inpaiting, semi-supervised learning, image colorization, we demonstrate that the weighted nonlocal total variation model has very good performance. It provides an effective and efficient method for many image processing and machine learning problem.

The rest of the paper is organized as follows. In section 1, we review the interpolation problem on point cloud, which is typically hard to solve by traditional interpolation method. Then the weighted nonlocal TV method (WNTV) is introduced in section 2

. We apply the split Bregman iteration algorithm to our method, which is a well-known algorithm to solve a very broad class of L1-regularization problems. Numerical experiments including semi-supervised clustering, image inpainting and image colorization are shown in section

3, 4 and 5 respectively. Here we compared our results to those obtained using graph Laplacian, nonlocal TV and weighted nonlocal Laplacian. Conclusions are made in the section 6.

## 2 Weighted Nonlocal TV

As introduced at the beginning of the introduction, we consider an interpolation problem in a high dimentional point cloud. Let be a set of points in and be a subset of . is a function on and with given . We assume that samples a smooth manifold embedded in and we want to minimize the total variation of on to solve on the whole poing cloud . This idea gives an optimization problem in continuous version:

 minu∫M|∇Mu(x)|dx,subject to:u(x)=g(x),x∈S, (6)

Using the nonlocal gradient in (2) to approximate the gradient , we have a discrete optimization problem

 min∑x∈V(∑y∈Vw(x,y)(u(x)−u(y))2)1/2,subject to:u(x)=g(x),x∈S, (7)

Inspired by the weighted nonlocal Laplacian method proposed by Shi et. al. in Shi2017 , we actually modify the above functional to add weight to balance the energy between labeled points and unlabeled sets:

 minu∑x∈V∖S(∑y∈Vω(x,y)(u(x)−u(y))2)1/2+|V||S|∑x∈S(∑y∈Vω(x,y)(u(x)−u(y))2)1/2, (8)

with the constraint

 u(x)=g(x),x∈S. (9)

where is a subset of the vertices set , and are the number of points in sets and , respectively. The idea is that when the sample rate is low, the summation over the unlabeled set overwhelms the summation over the labeled set such that the continuity on the labeled set is sacrificed. To maintain the continuity of the interpolation on the labeled points, we introduce a weight to balance the labeled term and the unlabeled term.

The weighted nonlocal total variation model (WNTV) (8) can be solved by split Bregman iteration doi:10.1137/080725891 . To simplify the notation, we introduce an operator as follows,

 DNGu(x,y)=⎧⎪ ⎪ ⎪⎨⎪ ⎪ ⎪⎩√ω(x,y)(u(x)−u(y)),if   x∈V∖S,|V||S|√ω(x,y)(u(x)−u(y)),if   x∈S.

With above operator, WNTV model (8) can be rewritten as

 minu,D∑x∈V(∑y∈V|D(x,y)|2)1/2,subject to:   D(x,y)=DNGu(x,y). (10)

with the constraint

 u(x)=g(x),x∈S.

We then use Bregman iteration to enforce the constraint to get a two-step iteration,

 (uk+1,Dk+1)=arg minu,D∑x∈V(∑y∈V|D(x,y)|2)1/2+λ2∑x∈V∑y∈V(D(x,y)−DNGu(x,y)−Qk(x,y))2, (11) subject to:u(x)=g(x),x∈S. Qk+1=Qk +(DNGuk+1−Dk+1). (12)

where is a positive parameter.

In above iteration, (12) is easy to solve. To solve the minimization problem (11), we use the idea in the split Bregman iteration to solve and alternatively.

 uk+1= argminu||Dk−DNGu−Qk||22,% subject to:u(x)=g(x),x∈S. (13) Dk+1= argminD∥D∥1+λ2||D−DNGuk+1−Qk||22. (14) Qk+1= Qk+(DNGuk+1−Dk+1). (15)

where

 ∥D∥1=∑x∈V(∑y∈V|D(x,y)|2)1/2.

The first step is a standard least-squares problem. It is staightforward to see that satisfies a linear system,

 ∑y∈V∖S(ω(x,y)+ω(y,x))(u(x)−u(y))+∑y∈S⎛⎝ω(x,y)+(|V||S|)2ω(y,x)⎞⎠(u(x)−u(y)) −∑y∈V∖S((Dk(x,y)−Qk(x,y))√ω(x,y)−(Dk(y,x)−Qk(y,x))√ω(y,x)) −∑y∈S((Dk(x,y)−Qk(x,y))√ω(x,y)−|V||S|(Dk(y,x)−Qk(y,x))√ω(y,x))=0,x∈V∖S, (16)

with the constraint

 u(x)=g(x),x∈S. (17)

The linear system (16)-(17) looks like complicated. Its coefficient matrix is sparse, symmetric and postive definite which can be solved efficiently by conjugate gradient method.

The minimizer of the optimization problem (14) can be explicitly computed using shrinkage operators doi:10.1137/080725891 . Notice that this problem is decoupled in terms of , i.e. actually solves a subproblem,

 minDx|Dx|+λ2||Dx−DNGxuk+1−Qkx||22,

where

 |Dx|=(∑y∈V|D(x,y)|2)1/2,

and .

It is well known that solution of above optimization problem can be given by soft shrinkage.

 Dk+1x=shrink(DNGxuk+1+Qkx,1/λ)

where

 shrink(z,γ)=z||z||2max(||z||2−γ,0)

Summarizing above discussion, we get an iterative algorithm to solve weighted nonlocal total variation model,

## 3 Semi-supervised Clustering

In this section, we test WNTV in a semi-supervised clustering problem on the famous MNIST data set mnist

. The MNIST database consists of 70,000 grayscale 28

28 pixel images of handwritten digits, see Fig. 1, which is divided into a training set of 60,000 examples, and a test set of 10,000 examples. The images include digits from 0 to 9, which can be viewed as 10 classes segmentation.

From geometrical point of view, 70,000 2828 images form a point cloud in 784-dimension Euclidean space. In the tests, we randomly select a small subset to label,

 S=l⋃iSi,

where is a subset of with label . Our task here is to label the rest of unlabeled images. The algorithm we used is summarized in Algorithm 2.

In our experiment of MNIST dataset, the weight function is then constructed using the Gaussian,

 ω(x,y)=exp(−∥x−y∥2σ(x)2),

where denotes the Euclidean distance, is the distance between and its 10th nearest neighbor. The weight is made sparse by setting equal to zero if point is not among the 20th closest points to point .

From the result of table (1), we can see that with high label rate (700/70000), all four methods give good classification. Nevertheless, as the label rate is reduced (100/70000, 50/70000), graph Laplacian and nonlocal TV both fail. The results given by WNTV and WNLL still have reasonable accuracy. WNTV is slightly better than WNLL in our tests.

## 4 Image Inpainting

The problem of fitting the missing pixels of a corrupted image is always of interest in image processing. This problem can be formulated as an interpolation problem on point cloud by considering patches of the image. Consider a discrete image , around each pixel , we define a patch that is collection of pixels of image . The collection of all patches is defined to be the patch set osher2016low ,

 P(f)={pij(f):(i,j)∈{1,2,...,m}×{1,2,...,n}}

Here forms a point set .

Then the image can be viewed as a function on the point cloud . is defined to be the intensity of the central pixel of the patch,

 u(pij(f))=f(i,j),

where is the intensity of pixel .

Now, given subsample of the image, the problem of image inpainting is to fit the missing value of on the patch set . However, this problem is actually more difficult than the interpolation, since the patches is also unknown. In the image inpainting, we also need to recover the point cloud in addition to the interpolation function. We achieve this by a simple iterative scheme. First, we fill in the missing pixels by random number to get a complete image. For this complete (quality is bad) image, we construct point cloud by extracting patches. On this point cloud, we run WNTV to compute an interpolation function. From this interpolation function, we can construct an image. Then the patch set is updated from this new image. By repeating this process until convergence, we get the restoration of the image. We summarize this ideas in algorithm (3).

### 4.1 Grayscale image inpainting

We first apply the algorithm to grayscale images. In this case, we also use Gaussian weight,

 ω(x,y)=exp(−∥x−y∥2σ(x)2)

where is the Euclidean distance between patches and . is the distance between and its 20th nearest neighbor. The weight is made sparse by setting equal to zero if point is not among the 50th closest points to point . For each pixel, we assign a 1111 patch around it consisting of intensity values of pixels in the patch. In order to accelerate the speed of convergence, we use the semi-local patch by adding the local coordinate to the end of the patches,

 pij(I)=[pij,λ1i,λ2j]

where

 λ1=3||fS||∞m,   λ2=3||fS||∞n.

An approximate nearest neighbor algorithm (ANN) is used to obtain nearest neighbors. We use the Peak Signal-to-Noise Ratio (PSNR) to measure the quality of restored images,

 PSNR(u,ugt)=−20log10(∥u−ugt∥/255)

where and are the restored image and the original image respectively.

The results are displayed in Fig. 2, 3 and 4. For each image, we fix the number of iterations to be 10. As we can see, WNTV and WNLL performs much better than classical nonlocal TV method and graph Laplacian. The results of WNLL are comparable to proposed WNTV. As expected, WNTV works better for cartoon image as shown in Fig. 4.

### 4.2 Color Image Inpainting

Now, we apply the algorithm to color images. The basic settings are similar to the grayscale image examples. In color image, patch becomes a 3D cube. The size we used is . We also use Gaussian weight,

 ω(x,y)=exp(−∥x−y∥2σ(x)2)

where is the Euclidean distance between patches and . is the distance between and its 20th nearest neighbor. The weight is made sparse by setting equal to zero if point is not among the 50th closest points to point . The color image is recovered in RGB channels separately.

We apply our algorithm to Fig. 5(a) and 6(a). Again, WNTV and WNLL outperform NTV and GL. In the image of house, in which cartoon dominates, the result of WNTV is better than WNLL. While in the image of Barbara, WNTV and WNLL are comparable since this image is rich in textures.

## 5 Image Colorization

Colorization is the process of adding color to monochrome images. It is usually done by person who is color expert but still this process is time consuming and sometimes could be boring. One way to reduce the working load is only add color in part of the pixels by human and using some colorization method to extend the color to other pixels.

This problem can be natrually formulated as an interpolation on point cloud. The point cloud is constructed by taking patches from the gray image. On the patches, we have three functions, , and corresponding to three channels of the color image. Then WNTV is used to interpolate , and over the whole patch set. The weight is computed in the same way as that in image inpainting.

The colorization results from 1% samples are demonstrated in Fig. 7, 8 and 9. Face of the baboon, snow mountains and wings of butterfly are not properly colored in GL and NTV. The face of baboon are blured. Part of the wings are colored in yellow by mistake. Snow mountain and text on F16 are also blured. In WNLL and WNTV, they are all properly colored. In addition, PSNR value also suggest that WNTV has the best performance.

## 6 Conclusion

In this paper, we propose a weighted nonlocal total variation (WNTV) model for interpolations on high dimensional point cloud. This model can be solved efficiently by split Bregman iteration. Numerical tests in semi-supervised learning, image inpainting and image colorization demonstrate that WNTV is an effective and efficient method in image processing and data analysis.

## References

• [1] A. Buades, B. Coll, and J.-M. Morel. A review of image denoising algorithms, with a new one. Multiscale Model. Simul., 4:490–530, 2005.
• [2] A. Buades, B. Coll, and J.-M. Morel. Neighborhood filters and pde’s. Numer. Math., 105:1–34, 2006.
• [3] F. R. K. Chung. Spectral Graph Theory. American Mathematical Society, 1997.
• [4] G. Gilboa and S. Osher. Nonlocal linear image regularization and supervised segmentation. Multiscale Model. Simul., 6:595–630, 2007.
• [5] G. Gilboa and S. Osher. Nonlocal operators with applications to image processing. Multiscale Model. Simul., 7:1005–1028, 2008.
• [6] T. Goldstein and S. Osher. The split bregman method for l1-regularized problems. SIAM Journal on Imaging Sciences, 2(2):323–343, 2009.
• [7] Y. LeCun, C. Cortes, and C. J. Burges. Mnist database.
• [8] S. Osher, M. Burger, D. Goldfarb, J. Xu, and W. Yin. An iterative regularization method for total variation-based image restoration. Multiscale Modeling & Simulation, 4(2):460–489, 2005.
• [9] S. Osher, Z. Shi, and W. Zhu. Low dimensional manifold model for image processing. Technical Report, CAM report 16-04, UCLA, 2016.
• [10] S. Osher, Z. Shi, and W. Zhu. Low dimensional manifold model for image processing. SIAM Journal on Imaging Sciences, 10(4):1669–1690, 2017.
• [11] L. I. Rudin, S. Osher, and E. Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena, 60(1-4):259–268, 1992.
• [12] Z. Shi, S. Osher, and W. Zhu. Weighted nonlocal laplacian on interpolation from sparse data. Journal of Scientific Computing, Apr 2017.
• [13] Z. Shi, J. Sun, and M. Tian. Harmonic extension on point cloud. arXiv:1509.06458.
• [14] X. Zhu, Z. Ghahramani, and J. D. Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In Machine Learning, Proceedings of the Twentieth International Conference ICML 2003), August 21-24, 2003, Washington, DC, USA, pages 912–919, 2003.