Efficient Regularization of Squared Curvature

11/07/2013 ∙ by Claudia Nieuwenhuis, et al. ∙ 0

Curvature has received increased attention as an important alternative to length based regularization in computer vision. In contrast to length, it preserves elongated structures and fine details. Existing approaches are either inefficient, or have low angular resolution and yield results with strong block artifacts. We derive a new model for computing squared curvature based on integral geometry. The model counts responses of straight line triple cliques. The corresponding energy decomposes into submodular and supermodular pairwise potentials. We show that this energy can be efficiently minimized even for high angular resolutions using the trust region framework. Our results confirm that we obtain accurate and visually pleasing solutions without strong artifacts at reasonable run times.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 3

page 4

page 6

page 8

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

A number of vision tasks can be formulated as an energy minimization problem such as segmentation, 3D reconstruction, stereo and inpainting. The corresponding energies typically consist of a data affiliation term and a regularization term. The data affiliation term relates the solution to the image data, while the regularization term imposes some kind of prior knowledge on the result. Length-based regularizers give rise to sub-modular energies (Potts model) that can be globally and efficiently optimized and, therefore, are widely used. Their main disadvantage is an inherent shrinking bias, which tends to eliminate thin and elongated structures such as vessels or limbs, see Figure 1(b). In contrast, curvature-based regularizers preserve such fine details, but are more difficult to model, often resulting in non-submodular energies that are hard to optimize.

Previous approaches to curvature regularization are computationally intensive [11, 13], have strong discretization artifacts [11, 13, 5, 7, 12] and are often restricted to specific angular resolutions [5] or grid complexes [11, 13, 7]. Curvature is often combined with length to compute Euler’s elastica, which can alleviate some of the problems of pure curvature regularizers, see Figure 1(c). However, satisfactory solutions to the curvature regularization problem have not been found yet.

In this paper we present a novel approach to modeling and efficiently optimizing squared curvature, which yields visually pleasing results without strong discretization artifacts, see Figure 1(d).

a) Original (b) Length Regularization
(c) Elastica Regularization [7] (d) Proposed
Figure 1: Segmentation results of Picasso’s ink drawing using (b) length based regularization computed with the Potts model based on a 16 neighborhood using [2], (c) Euler’s elastica consisting of squared curvature and length regularization computed by Heber et al. [7], (d) the novel squared curvature regularizer.

Our goal is to regularize the boundary of a binary labeling based on an integral of squared curvature

(1)

For example, a segmentation energy can combine this regularization term with a regional appearance term for some potential function over pixels .

We optimize a binary energy function with triple clique potentials, which can be reduced to an integer quadratic energy with both submodular and supermodular pairwise terms without adding auxiliary variables.

Such energies can be efficiently optimized with the LSA-TR method proposed in our companion paper [6]. Our formalism is based on integral geometry, allows for high angular resolutions and yields excellent results compared to previous approaches.

Our contributions can be summarized as follows

  • We propose a novel model for measuring squared curvature based on integral geometry and show how it relates to counting straight triple cliques.

  • Our model can be formulated as a pairwise quadratic energy and optimized efficiently even for high angular resolutions.

  • The proposed approach outperforms previous methods in terms of quality of the results as well as efficiency.

Related work: Most existing models for curvature regularization in vision are based on the Bruckstein formula [11, 13, 7, 5] except for [12] which is based on learning.

Shekhovtsov et al. [12] learn the costs for curvature in the form of “soft” patterns which serve as filters whose response is locally minimized. However, their MRF approach lacks accuracy due to missing consistency constraints between neighboring patterns.

Approaches based on the Bruckstein formula [4] express curvature as the exterior angle sum of an approximating polygon. Mostly this is done by formulating an optimization problem on a cell complex, which is a planar graph with fixed, regular structure. A segmentation is a consistent subset of faces and edges. Local “curvature” is then measured by the exterior angle between consecutive boundary edges.

The pioneering approach to image segmentation with curvature regularization on cell complexes was given by Schoenemann et al. [11]

. They solve a linear program where each variable corresponds to a configuration of two edges with associated angle. Strandmark and Kahl

[13] improve this framework by removing extraneous arcs and generalize it to 3D surfaces. Heber et al. [7] provide a formulation of the curvature model on cell complexes that can be optimized by approximating the envelope of the underlying non-convex functional.

The main drawback of the approaches formulated on cell complexes is their high runtime up to several hours, which is due to the large number of variables and consistency constraints. In addition, these methods suffer from a strong angular bias since they only allow for specific edge configurations. Although angular resolution can in theory be increased arbitrarily, solving the problem for high resolutions easily becomes infeasible. This is not the case for our approach since the number of triple cliques grows linearly with the angular resolution.

Another approach based on the Bruckstein formula which is related to cell complexes is El-Zehiry and Grady’s work [5]. They formulate their problem on a regular pixel grid, which is interpreted as a cell complex. Accordingly, their angular resolution is limited by 90 degrees, which leads to a coarse approximation of curvature. Extending this approach to a higher angular resolution is not possible due to inconsistencies with the Bruckstein formula.

Our method differs from the previous approaches in that we neither explicitly model our segmentation boundary as a polygon by means of cell complexes, nor do we measure angles between edges. Instead, we count the number of “active” straight line triple cliques and relate it to squared curvature based on integral geometry.

2 Integrating Squared Curvature

We propose a new discrete model for approximating the squared curvature integral in (1) based on a certain class of triple cliques. Our combinatorial approach is presented below in the context of 2D segmentation and inpainting. It has straightforward extensions to segmentation problems in higher dimensions, but this is left for future work, see Section 5. Unlike previous discrete

methods for curvature, we use a neighborhood system of symmetric ’straight’ triple cliques. Our general intuition is illustrated on the right. Let the indicator variables on pixels define the segment and consider a collection of triple cliques for each pixel and its symmetric neighbors for fixed 2D shifts within a neighborhood. The colors show triple cliques with binary configurations or at different pixels . Such triple cliques ’fire’ only on curved parts of the boundary of . The number of responses increases for larger curvature. Configurations respond to positive and to negative curvature. These observations suggest that such triple cliques can measure curvature. The details are presented below.

2.1 Notation: Variables, Cliques, Neighborhood

The proposed neighborhood structure is somewhat similar to the standard regular neighborhood of pairwise cliques (edges) commonly used for length-based regularization [3]. At each pixel the neighborhood is bounded, e.g. by a box centered at this pixel, see Figure 2(b). Interactions are imposed between pixel and its neighbors in a certain discrete set of directions limited by the grid locations within the box. Standard length-based regularization [3] corresponds to pairwise Potts interactions imposed over and each neighbor where denotes a shift of given length in direction . Our new curvature-based regularization model uses interaction potentials imposed over and pairs of symmetric neighbors in directions and forming a straight triple clique

(2)

centered at pixel , see Figure 2

(b). We will use standard binary variables

to denote the object/background label at pixel

. Vector

will denote a configuration of binary labels of all pixels in defining segment . Vector will denote a configuration of labels for pixels in clique .

Each triple clique centered at pixel is described by its orientation and distance , see Figure 2(b). We will also use to denote the angular rotation to the next clique. Since our triple cliques include symmetric pairs of neighbors, the actual number of distinct triple cliques centered at pixel is half the number of its neighbors. To avoid confusion, in the rest of the paper denotes the number of distinct triple cliques at pixel where is an index of orientation in the set such that . This avoids identical cliques with orientations and .

2.2 Partial Sum Approximation of (1)

(a) Contour points (b) 3-clique of orientation
Figure 2: The partial sum approximation (3) of the integral uses a sequence of points on the contour . Our approach (a) is to select all contour points where the tangent is consistent with the orientation of some triple clique in a given neighborhood (b). For example, (a) marks two points with a tangent of the same orientation as clique in (b). Each point corresponds to a particular clique index .
Figure 3: Examples of triple cliques of orientation that “fire” in a vicinity of point on contour in Figure 2(a). The brown region highlights points where clique has configuration . The area of this region in (13), that is the number of such triple cliques, is determined by local curvature and by clique size parameter .

Our discrete model for evaluating the integral of squared curvature (1) is based on a partial sum approximation. Without loss of generality, assume that the (continuous) segment has genus zero and that its boundary is a closed contour , as shown in Figure 2(a). Then, the integral in (1) can be approximated as

(3)

where is a finite sequence of points , is a contour length between adjacent points, and is the curvature at point . In general, such approximations converge for finer discretizations as if the contour is sufficiently smooth.

It is common to select a sequence of approximating points at equal intervals. Our approach selects these points differently. We choose points where tangents coincide with orientations for a chosen neighborhood system, as illustrated in Figure 2(a,b). Assume that curvature is nearly constant between adjacent points, then where is the radius of the osculating circle at point and is the tangent orientation at point . Thus, the partial sum (3) becomes

(4)
Theorem 1.

Let contour point have tangent orientation and osculating ball of radius . Then, the set of all points such that and for given distance has area , see brown region in Figure 3.

This theorem is proved in the Appendix (Section 6

). It allows to accurately estimate curvature

at contour point using triple cliques of orientation consistent with the tangent at point . Assuming , Theorem 1 implies that the number of triple cliques in a vicinity of point with configuration is

which is the brown area in Figure 3. In case , the same number corresponds to configurations .

Assuming that triple cliques of configurations or have penalties depending on orientation , the overall cost of all such cliques is

where

(5)

is an indicator function. Choosing penalty

(6)

converts the right hand side in the equation above into

In combination with (3) and (4) this demonstrates that

(7)

which is our main technical result concluding this section.

Figure 4: Accuracy of our squared curvature model. The proposed estimated squared curvature (vertical axis) is computed on a circle of decreasing total curvature (i.e. increasing radius, horizontal axis) based on increasing clique size . To approximate large curvature small clique sizes are required, whereas large clique sizes are more accurate to measure large curvature and yield less noisy measurements. For curvature values we underestimate curvature, since in these cases the circle fits inside the neighborhood. This limits the estimated curvature energy by the number of pixels in the circle, i.e. its area, which grows quadratically.

2.3 Accuracy of Our Curvature Model

To demonstrate the accuracy of the proposed squared curvature model we generated a sequence of circles with decreasing total squared curvature (increasing radius ). We compared the computed curvature energy (7) obtained with different neighborhood sizes to the correct integral of squared curvature of the circle, . The plots in Figure 4 show that we indeed approximate squared curvature. We can also conclude that for smaller neighborhoods we more accurately measure larger curvature values but obtain more noisy results. In contrast, our results are less noisy for larger neighborhood masks but yield less accurate measurements for larger curvature. This is because our measurement is limited by the area of the circle, which is very small for small radius (large curvature) and grows quadratically. These findings are supported by Figure 5, which shows the contribution of each pixel to the measured curvature energy. To alleviate the issue of underestimating curvature for larger values of the image can be scaled to subpixel accuracy according to the increase in neighborhood size or stronger regularization can be used, see Section 4.1 in the experiments.

(a) (b) (c)
Figure 5: Response images showing for each pixel of a circle of increasing radius (i.e. decreasing squared curvature) its contribution to the curvature energy, see Figure 13. The computations are based on a clique size . The images show that the brown area in Figure 3 is approximated with our approach. (a) also shows that large curvature is underestimated for large values of (compare Figure 4).

3 Optimization

In (7) we derived our new squared curvature model, which can be used as a regularization energy

(8)

Let denote the three pixels corresponding to a triple clique given in (2). The function defined in (5) assigns each triple clique the value 1 if its configuration is (1,0,1) or (0,1,0). The corresponding triple clique energy in (8) naturally decomposes into a unary term, two pairwise submodular terms and a pairwise supermodular term without additional auxiliary variables

(9)

In general, energies of lower order are considered to be easier to optimize. This is an additional benefit of our integral geometry model that we get for ’free’.

Our energy is nevertheless non-submodular and, therefore, cannot be optimized globally. We use a new method called Local Submodular Approximations with Trust Region (LSA-TR) proposed in our companion paper [6]. The method is efficient and obtains state-of-the-art results that are very accurate on a wide range of applications, which we show in our companion paper.

We compare our results in terms of accuracy and runtime to other optimization approaches (TRW-S [8], LBP [9], QPBO-I [10] - an extension of QPBO [1]) in the experimental section 4.4. Throughout all experiments we use the same set of default parameters for LSA-TR.

Neighborhood Definition It remains to construct the neighborhood for defining our triple cliques. We use a fixed box of size for constructing triple cliques of size for a set of orientations around the central pixel (we denote the specific case of only horizontal and vertical neighbors in a neighborhood by “” throughout the experimental section). Neighbors are moved as far towards the edge of the box as possible, see Figure 6. The angle difference between neighboring triple cliques can then be computed as the angle between subsequent cliques in this neighborhood system. The corresponding weights are then given by (6).

Figure 6: Defining neighborhood systems of different sizes. For each we define a box of size . For each orientation in this neighborhood we then push the neighbor as far towards the boundary of the box as possible (see the red points, which are different from standard neighborhoods) to obtain the approximate distance of each neighbor to the central clique point. Triplets are then composed of opposite neighbors and the central point of the neighborhood. On the left we show an example for corresponding to a neighborhood, on the right for corresponding to a neighborhood.

4 Experiments

This section evaluates our squared curvature approach and compares our results to previous methods for curvature regularization.

We show results for binary segmentation and inpainting which minimize the following energy

(10)

The data term depends on the pixel color of the i-th pixel in the image . The weight

balances the impact of the regularizer with respect to the appearance term. We use a Gaussian of variance 0.4 for modeling the foreground and background data term in our experiments. The mean values are 0 and 0.6 for foreground and background respectively for the camera man examples and 0 and 1 for the Don Quixote image. For inpainting, we use the same energy but set the data term to 1 for foreground and background simultaneously for the region to be inpainted.

4.1 Triple Clique Length

We first show the effect of larger triple cliques in Figure 7. The larger the length of the triple cliques the more we underestimate large curvature values (see Figure 4) leading to noise in the segmentation. Yet, we would like to use large neighborhoods to attain a larger angular resolution of our clique orientations as shown in the next section. As a remedy we propose to scale the image according to triple clique length in order to achieve subpixel accuracy. In this way we do not underestimate large curvature and avoid angular artifacts. Alternatively, one can increase regularization to reduce the noise but this might lead to less details in the segmentation. These points are illustrated in Figure 7.

(a) , , 1x (b) , , 1x
(c) , , 3x (d) , , 1x
Figure 7: Segmentation results on the cameraman image showing the effect of using large triple cliques. (a) Segmentation result with novel squared curvature regularizer using a neighborhood (8-connectivity) (). (b) The same segmentation using a neighborhood (16-connectivity). Note the noise which is due to underestimating large curvature locally (see text for details). (c) By scaling the image (and adapting accordingly), we avoid underestimation while taking advantage of a bigger angular resolution. (d) Similar effects but with less details can be achieved by increasing the regularizer weight instead of scaling the image.

4.2 Angular Resolution

We now show the effect of increasing angular resolution on the segmentation results. Figure 8 shows results for angular resolutions of 90 degress ( neighborhood), 45 degrees () and 12.5 degrees (). For small curvature weight the appearance is strong yielding comparable results for all angular resolutions. For very large curvature weight block structures become apparent. This is due to the fact that the algorithm is blind to curvature for points whose tangent direction is not in our clique set. Accordingly, for only 90 degree resolution horizontal and vertical edges become dominant in the result. For 45 degree resolution diagonal edges appear as well. For larger resolution we obtain smooth boundaries.

, small , small , small
, large , large , large
Figure 8: Segmentation results for Picasso’s ink drawing for small and large curvature regularization weight and increasing neighborhood size. Note that the choice of the neighborhood size only becomes important for larger regularization weights. The images clearly show how ’blocky’ structures disappear with larger neighborhood sizes.

4.3 Inpainting Results

In this section we show results for the task of binary inpainting demonstrating that our approach is able to reconstruct shape with minimal curvature instead of length. We show two examples in Figure 9, for which we mask larger portions of the image for which we “occlude” its data: the data term in these regions is set to a constant value for both foreground and background. For the remaining part of the image we use a data term based on a Gaussian intensity model with fixed mean and variance for foreground and background. Figure 9 shows the results of our experiments and a comparison to length based inpainting. The results clearly show that we minimize the curvature of the object boundary, while length finds the shortest connection between boundary segments.

a) Input with mask b) Length c) Curvature
Figure 9: Results for binary inpainting with the proposed curvature regularizer compared to length based inpainting. a) Original image with masked regions to be inpainted, b) Result of length based inpainting, c) Result of curvature based inpainting with the novel approach.

4.4 Optimization and Efficiency

We evaluated the performance and runtime of different optimization strategies for minimizing the energy in (10). In particular, we compared QPBO-I [10], TRW-S [8], Loopy Belief Propagation (LBP) [9] and LSA-TR [6]. For TRW-S we stopped computations after 50,000 steps without convergence of the algorithm. Running both TRW-S and LBP even for 500,000 steps did not improve the results. Figure 10 shows the energies we obtain for the respective methods on the vertical axis plotted against the regularization weight on the horizontal axis. Of all the tested algorithms, LSA-TR finds the lowest energy for our problem for almost all values . This is especially true for higher curvature weights, where QPBO-I and TRW-S compute trivial solutions of higher energy with almost all pixels labeled as background. In particular, QPBO is unable to label any pixels at all for and therefore QPBO-I cannot improve the result either. The high energies we get with LBP for correlates with the extremely noisy results the algorithm returns. Figure 11 shows a comparison in runtime revealing that TRW-S is least efficient of the four algorithms, whereas LSA-TR is the fastest optimization method. The proposed method also compares remarkably well to other approaches that compute squared curvature. For the results in Figure 12 the runtimes are as follows: Heber et al. [7] 1 to 5 minutes, Schoenemann et al. [11] 10 minutes to 3.5 hours and only El-Zehiry and Grady’s method is fast with 10 seconds per image. Strandmark and Kahl’s approach [13] is only slightly faster than Schoenemann’s.

Figure 10: Comparison of optimized energies based on different regularization strengths for the following optimization algorithms: LSA-TR [6], TRW-S [8], LBP [9], QPBO-I [10]. LSA-TR clearly outperforms the other algorithms yielding reasonable results even for strong regularization, while the other approaches break down with trivial (TRWS, QPBO-I) or extremely noisy solutions (LBP).
QPBO TRW-S LBP LSA-TR (7x7)
0.05 0.7 (2%) 751 680 0.8 0.7
0.1 9.3 (30%) 764 682 1.4 1.6
0.5 15.2 (100%) 715 709 3.9 9.2
1 15.4 (100%) 713 795 10.6 48.2
5 15.3 (100%) 714 794 16.5 88
10 15.4 (100%) 703 797 7.9 110
Figure 11: Runtime comparison for different optimization methods ( neighborhood). For QPBO [1] the percentage of unlabeled pixels is indicated in parenthesis. For TRW-S we stopped computations after 50,000 steps without convergence. LSA-TR is also listed for a neighborhood with to obtain comparable results.

4.5 Comparison to Other Approaches

We compare our curvature regularizer to previous approaches in Figure 12 where we show results for small and large regularization for the approach by Heber et al. [7], Schoenemann et al. [11], Strandmark and Kahl [13] and El-Zehiry and Grady [5]. For all of these methods strong artifacts are evident, which become worse for stronger regularization. Heber et al. [7] compute elastica (i.e. length and squared curvature regularization), which preserves long structures, but also introduces block artifacts. El-Zehiry and Grady are limited by 90 degree resolution, i.e. their results are composed of blocks and contain large regions of unlabeled pixels QPBO [1] did not label. Schoenemann et al. [11] obtain block artifacts with increasing regularization. Strandmark and Kahl [13] do not preserve fine details and also suffer from some angular resolution artifacts. In contrast, our method (neighborhood size ) preserves fine details in the segmentation and produces clean object boundaries for weak and strong regularization without requiring excessive runtimes.

(a) original (b) [7] small (c) [7] large (d) [5] small (e) [5] large (f) ours small (g) ours large
(h) original (i) [11] small (j) [11] large (k) [13] small (l) [13] large (m) ours small (n) ours large
Figure 12: Comparison of our results for 7x7 neighborhood to previous curvature regularizers for smaller and larger curvature weight, (b+c) elastica by Heber et al. [7], (d+e) El-Zehiry and Grady [5], (i+j)Schoenemann et al. [11], (k+l) Strandmark and Kahl [13]. Grey pixels are unassigned, which is due to QPBO leaving pixels unlabeled in El-Zehiry and Grady [5] - using QPBO-I does not improve the results.

5 Conclusion and Future Work

In this paper we proposed a novel approach to squared curvature computation and regularization. We gave an integral geometric derivation of our method which justifies that straight triple cliques can be used to measure squared curvature. Our triple cliques decompose into a set of submodular and supermodular pairwise cliques, which can be efficiently optimized by LSA-TR. We showed that our approach works for high angular resolutions and thus does not suffer from grid artifacts as do previous methods. The results demonstrate that we outperform these methods in terms of quality and efficiency.
This approach naturally extends to higher dimensions for the regularization of 3D surfaces, which will be important e.g. for 3D reconstruction approaches. We will leave this for future work.

6 Appendix: Proof of Theorem 1

In this appendix we will prove Theorem 1. We will use an integral geometric argument to show that squared curvature is related to the brown area in Figure 3.

(a) (b)
Figure 13: (a) Computing the partial area of a circle above a half chord of length , (b) Computing the shaded area

We first state that the partial area of the circle of curvature above a half chord of length in Figure 13(a) is given by

(11)

Based on this expression we can now derive the area in Figure 13(b) with respect to a half chord of length in a circle of radius

(12)

Using Taylor approximation w.r.t. we obtain

(13)

References

  • [1] E. Boros, P. Hammer, and X. Sun. Network flows and minimization of quadratic pseudo-boolean functions. Technical report RRR 17-1991, RUTCOR, 1991.
  • [2] Y. Boykov and M.-P. Jolly. Interactive graph cuts for optimal boundary & region segmentation of objects in n-d images. In Int. Conf. of Computer Vision, Vancover, Canada, 2001.
  • [3] Y. Boykov and V. Kolmogorov. Computing geodesics and minimal surfaces via graph cuts. In International Conference on Computer Vision (ICCV), 2003.
  • [4] A. Bruckstein, A. Netravali, and T. Richardson. Epi-convergence of discrete elastica. Applicable Analysis, 79(1-2):137–171, 2001.
  • [5] N. El-Zehiry and L. Grady. Fast global optimization of curvature. In

    Conf. Computer Vision and Pattern Recognition

    , 2010.
  • [6] L. Gorelick, Y. Boykov, O. Veksler, I. BenAyed, and A. Delong. Submodularization for Quadratic Pseudo-Boolean Optimization. In ArXiv, 2013.
  • [7] S. Heber, R. Ranftl, and T. Pock. Approximate envelope minimization for curvature regularity. In Proceedings of the 12th international conference on Computer Vision - Volume Part III, European Conf. on Computer Vision, pages 283–292, 2012.
  • [8] V. Kolmogorov. Convergent tree-reweighted message passing for energy minimization. IEEE Transanctions on Pattern Analysis and Machine. Intelligence., 28:1568–1583, October 2006.
  • [9] J. Pearl. Reverend bayes on inference engines: A distributed hierarchical approach. In

    National Conference on Artificial Intelligence

    , pages 133–136, 1982.
  • [10] C. Rother, V. Kolmogorov, V. Lempitsky, and M. Szummer. Optimizing binary mrfs via extended roof duality. In Conf. Computer Vision and Pattern Recognition, pages 1–8, 2007.
  • [11] T. Schoenemann, F. Kahl, S. Masnou, and D. Cremers. A linear framework for region-based image segmentation and inpainting involving curvature penalization. Int. Journal of Computer Vision, 2012.
  • [12] A. Shekhovtsov, P. Kohli, and C. Rother. Curvature prior for MRF-based segmentation and shape inpaint. In DAGM, 2012.
  • [13] P. Strandmark and F. Kahl. Curvature regularization for curves and surfaces in a global optimization framework. In EMMCVPR, pages 205–218, 2011.