A number of vision tasks can be formulated as an energy minimization problem such as segmentation, 3D reconstruction, stereo and inpainting. The corresponding energies typically consist of a data affiliation term and a regularization term. The data affiliation term relates the solution to the image data, while the regularization term imposes some kind of prior knowledge on the result. Length-based regularizers give rise to sub-modular energies (Potts model) that can be globally and efficiently optimized and, therefore, are widely used. Their main disadvantage is an inherent shrinking bias, which tends to eliminate thin and elongated structures such as vessels or limbs, see Figure 1(b). In contrast, curvature-based regularizers preserve such fine details, but are more difficult to model, often resulting in non-submodular energies that are hard to optimize.
Previous approaches to curvature regularization are computationally intensive [11, 13], have strong discretization artifacts [11, 13, 5, 7, 12] and are often restricted to specific angular resolutions  or grid complexes [11, 13, 7]. Curvature is often combined with length to compute Euler’s elastica, which can alleviate some of the problems of pure curvature regularizers, see Figure 1(c). However, satisfactory solutions to the curvature regularization problem have not been found yet.
In this paper we present a novel approach to modeling and efficiently optimizing squared curvature, which yields visually pleasing results without strong discretization artifacts, see Figure 1(d).
|a) Original||(b) Length Regularization|
|(c) Elastica Regularization ||(d) Proposed|
Our goal is to regularize the boundary of a binary labeling based on an integral of squared curvature
For example, a segmentation energy can combine this regularization term with a regional appearance term for some potential function over pixels .
We optimize a binary energy function with triple clique potentials, which can be reduced to an integer quadratic energy with both submodular and supermodular pairwise terms without adding auxiliary variables.
Such energies can be efficiently optimized with the LSA-TR method proposed in our companion paper . Our formalism is based on integral geometry, allows for high angular resolutions and yields excellent results compared to previous approaches.
Our contributions can be summarized as follows
We propose a novel model for measuring squared curvature based on integral geometry and show how it relates to counting straight triple cliques.
Our model can be formulated as a pairwise quadratic energy and optimized efficiently even for high angular resolutions.
The proposed approach outperforms previous methods in terms of quality of the results as well as efficiency.
Shekhovtsov et al.  learn the costs for curvature in the form of “soft” patterns which serve as filters whose response is locally minimized. However, their MRF approach lacks accuracy due to missing consistency constraints between neighboring patterns.
Approaches based on the Bruckstein formula  express curvature as the exterior angle sum of an approximating polygon. Mostly this is done by formulating an optimization problem on a cell complex, which is a planar graph with fixed, regular structure. A segmentation is a consistent subset of faces and edges. Local “curvature” is then measured by the exterior angle between consecutive boundary edges.
The pioneering approach to image segmentation with curvature regularization on cell complexes was given by Schoenemann et al. 
. They solve a linear program where each variable corresponds to a configuration of two edges with associated angle. Strandmark and Kahl improve this framework by removing extraneous arcs and generalize it to 3D surfaces. Heber et al.  provide a formulation of the curvature model on cell complexes that can be optimized by approximating the envelope of the underlying non-convex functional.
The main drawback of the approaches formulated on cell complexes is their high runtime up to several hours, which is due to the large number of variables and consistency constraints. In addition, these methods suffer from a strong angular bias since they only allow for specific edge configurations. Although angular resolution can in theory be increased arbitrarily, solving the problem for high resolutions easily becomes infeasible. This is not the case for our approach since the number of triple cliques grows linearly with the angular resolution.
Another approach based on the Bruckstein formula which is related to cell complexes is El-Zehiry and Grady’s work . They formulate their problem on a regular pixel grid, which is interpreted as a cell complex. Accordingly, their angular resolution is limited by 90 degrees, which leads to a coarse approximation of curvature. Extending this approach to a higher angular resolution is not possible due to inconsistencies with the Bruckstein formula.
Our method differs from the previous approaches in that we neither explicitly model our segmentation boundary as a polygon by means of cell complexes, nor do we measure angles between edges. Instead, we count the number of “active” straight line triple cliques and relate it to squared curvature based on integral geometry.
2 Integrating Squared Curvature
We propose a new discrete model for approximating the squared curvature integral in (1) based on a certain class of triple cliques. Our combinatorial approach is presented below in the context of 2D segmentation and inpainting. It has straightforward extensions to segmentation problems in higher dimensions, but this is left for future work, see Section 5. Unlike previous discrete
methods for curvature, we use a neighborhood system of symmetric ’straight’ triple cliques. Our general intuition is illustrated on the right. Let the indicator variables on pixels define the segment and consider a collection of triple cliques for each pixel and its symmetric neighbors for fixed 2D shifts within a neighborhood. The colors show triple cliques with binary configurations or at different pixels . Such triple cliques ’fire’ only on curved parts of the boundary of . The number of responses increases for larger curvature. Configurations respond to positive and to negative curvature. These observations suggest that such triple cliques can measure curvature. The details are presented below.
2.1 Notation: Variables, Cliques, Neighborhood
The proposed neighborhood structure is somewhat similar to the standard regular neighborhood of pairwise cliques (edges) commonly used for length-based regularization . At each pixel the neighborhood is bounded, e.g. by a box centered at this pixel, see Figure 2(b). Interactions are imposed between pixel and its neighbors in a certain discrete set of directions limited by the grid locations within the box. Standard length-based regularization  corresponds to pairwise Potts interactions imposed over and each neighbor where denotes a shift of given length in direction . Our new curvature-based regularization model uses interaction potentials imposed over and pairs of symmetric neighbors in directions and forming a straight triple clique
centered at pixel , see Figure 2
(b). We will use standard binary variablesto denote the object/background label at pixel
. Vectorwill denote a configuration of binary labels of all pixels in defining segment . Vector will denote a configuration of labels for pixels in clique .
Each triple clique centered at pixel is described by its orientation and distance , see Figure 2(b). We will also use to denote the angular rotation to the next clique. Since our triple cliques include symmetric pairs of neighbors, the actual number of distinct triple cliques centered at pixel is half the number of its neighbors. To avoid confusion, in the rest of the paper denotes the number of distinct triple cliques at pixel where is an index of orientation in the set such that . This avoids identical cliques with orientations and .
2.2 Partial Sum Approximation of (1)
|(a) Contour points||(b) 3-clique of orientation|
Our discrete model for evaluating the integral of squared curvature (1) is based on a partial sum approximation. Without loss of generality, assume that the (continuous) segment has genus zero and that its boundary is a closed contour , as shown in Figure 2(a). Then, the integral in (1) can be approximated as
where is a finite sequence of points , is a contour length between adjacent points, and is the curvature at point . In general, such approximations converge for finer discretizations as if the contour is sufficiently smooth.
It is common to select a sequence of approximating points at equal intervals. Our approach selects these points differently. We choose points where tangents coincide with orientations for a chosen neighborhood system, as illustrated in Figure 2(a,b). Assume that curvature is nearly constant between adjacent points, then where is the radius of the osculating circle at point and is the tangent orientation at point . Thus, the partial sum (3) becomes
Let contour point have tangent orientation and osculating ball of radius . Then, the set of all points such that and for given distance has area , see brown region in Figure 3.
This theorem is proved in the Appendix (Section 6
). It allows to accurately estimate curvatureat contour point using triple cliques of orientation consistent with the tangent at point . Assuming , Theorem 1 implies that the number of triple cliques in a vicinity of point with configuration is
which is the brown area in Figure 3. In case , the same number corresponds to configurations .
Assuming that triple cliques of configurations or have penalties depending on orientation , the overall cost of all such cliques is
is an indicator function. Choosing penalty
converts the right hand side in the equation above into
which is our main technical result concluding this section.
2.3 Accuracy of Our Curvature Model
To demonstrate the accuracy of the proposed squared curvature model we generated a sequence of circles with decreasing total squared curvature (increasing radius ). We compared the computed curvature energy (7) obtained with different neighborhood sizes to the correct integral of squared curvature of the circle, . The plots in Figure 4 show that we indeed approximate squared curvature. We can also conclude that for smaller neighborhoods we more accurately measure larger curvature values but obtain more noisy results. In contrast, our results are less noisy for larger neighborhood masks but yield less accurate measurements for larger curvature. This is because our measurement is limited by the area of the circle, which is very small for small radius (large curvature) and grows quadratically. These findings are supported by Figure 5, which shows the contribution of each pixel to the measured curvature energy. To alleviate the issue of underestimating curvature for larger values of the image can be scaled to subpixel accuracy according to the increase in neighborhood size or stronger regularization can be used, see Section 4.1 in the experiments.
In (7) we derived our new squared curvature model, which can be used as a regularization energy
Let denote the three pixels corresponding to a triple clique given in (2). The function defined in (5) assigns each triple clique the value 1 if its configuration is (1,0,1) or (0,1,0). The corresponding triple clique energy in (8) naturally decomposes into a unary term, two pairwise submodular terms and a pairwise supermodular term without additional auxiliary variables
In general, energies of lower order are considered to be easier to optimize. This is an additional benefit of our integral geometry model that we get for ’free’.
Our energy is nevertheless non-submodular and, therefore, cannot be optimized globally. We use a new method called Local Submodular Approximations with Trust Region (LSA-TR) proposed in our companion paper . The method is efficient and obtains state-of-the-art results that are very accurate on a wide range of applications, which we show in our companion paper.
We compare our results in terms of accuracy and runtime to other optimization approaches (TRW-S , LBP , QPBO-I  - an extension of QPBO ) in the experimental section 4.4. Throughout all experiments we use the same set of default parameters for LSA-TR.
Neighborhood Definition It remains to construct the neighborhood for defining our triple cliques. We use a fixed box of size for constructing triple cliques of size for a set of orientations around the central pixel (we denote the specific case of only horizontal and vertical neighbors in a neighborhood by “” throughout the experimental section). Neighbors are moved as far towards the edge of the box as possible, see Figure 6. The angle difference between neighboring triple cliques can then be computed as the angle between subsequent cliques in this neighborhood system. The corresponding weights are then given by (6).
This section evaluates our squared curvature approach and compares our results to previous methods for curvature regularization.
We show results for binary segmentation and inpainting which minimize the following energy
The data term depends on the pixel color of the i-th pixel in the image . The weight
balances the impact of the regularizer with respect to the appearance term. We use a Gaussian of variance 0.4 for modeling the foreground and background data term in our experiments. The mean values are 0 and 0.6 for foreground and background respectively for the camera man examples and 0 and 1 for the Don Quixote image. For inpainting, we use the same energy but set the data term to 1 for foreground and background simultaneously for the region to be inpainted.
4.1 Triple Clique Length
We first show the effect of larger triple cliques in Figure 7. The larger the length of the triple cliques the more we underestimate large curvature values (see Figure 4) leading to noise in the segmentation. Yet, we would like to use large neighborhoods to attain a larger angular resolution of our clique orientations as shown in the next section. As a remedy we propose to scale the image according to triple clique length in order to achieve subpixel accuracy. In this way we do not underestimate large curvature and avoid angular artifacts. Alternatively, one can increase regularization to reduce the noise but this might lead to less details in the segmentation. These points are illustrated in Figure 7.
|(a) , , 1x||(b) , , 1x|
|(c) , , 3x||(d) , , 1x|
4.2 Angular Resolution
We now show the effect of increasing angular resolution on the segmentation results. Figure 8 shows results for angular resolutions of 90 degress ( neighborhood), 45 degrees () and 12.5 degrees (). For small curvature weight the appearance is strong yielding comparable results for all angular resolutions. For very large curvature weight block structures become apparent. This is due to the fact that the algorithm is blind to curvature for points whose tangent direction is not in our clique set. Accordingly, for only 90 degree resolution horizontal and vertical edges become dominant in the result. For 45 degree resolution diagonal edges appear as well. For larger resolution we obtain smooth boundaries.
|, small||, small||, small|
|, large||, large||, large|
4.3 Inpainting Results
In this section we show results for the task of binary inpainting demonstrating that our approach is able to reconstruct shape with minimal curvature instead of length. We show two examples in Figure 9, for which we mask larger portions of the image for which we “occlude” its data: the data term in these regions is set to a constant value for both foreground and background. For the remaining part of the image we use a data term based on a Gaussian intensity model with fixed mean and variance for foreground and background. Figure 9 shows the results of our experiments and a comparison to length based inpainting. The results clearly show that we minimize the curvature of the object boundary, while length finds the shortest connection between boundary segments.
|a) Input with mask||b) Length||c) Curvature|
4.4 Optimization and Efficiency
We evaluated the performance and runtime of different optimization strategies for minimizing the energy in (10). In particular, we compared QPBO-I , TRW-S , Loopy Belief Propagation (LBP)  and LSA-TR . For TRW-S we stopped computations after 50,000 steps without convergence of the algorithm. Running both TRW-S and LBP even for 500,000 steps did not improve the results. Figure 10 shows the energies we obtain for the respective methods on the vertical axis plotted against the regularization weight on the horizontal axis. Of all the tested algorithms, LSA-TR finds the lowest energy for our problem for almost all values . This is especially true for higher curvature weights, where QPBO-I and TRW-S compute trivial solutions of higher energy with almost all pixels labeled as background. In particular, QPBO is unable to label any pixels at all for and therefore QPBO-I cannot improve the result either. The high energies we get with LBP for correlates with the extremely noisy results the algorithm returns. Figure 11 shows a comparison in runtime revealing that TRW-S is least efficient of the four algorithms, whereas LSA-TR is the fastest optimization method. The proposed method also compares remarkably well to other approaches that compute squared curvature. For the results in Figure 12 the runtimes are as follows: Heber et al.  1 to 5 minutes, Schoenemann et al.  10 minutes to 3.5 hours and only El-Zehiry and Grady’s method is fast with 10 seconds per image. Strandmark and Kahl’s approach  is only slightly faster than Schoenemann’s.
4.5 Comparison to Other Approaches
We compare our curvature regularizer to previous approaches in Figure 12 where we show results for small and large regularization for the approach by Heber et al. , Schoenemann et al. , Strandmark and Kahl  and El-Zehiry and Grady . For all of these methods strong artifacts are evident, which become worse for stronger regularization. Heber et al.  compute elastica (i.e. length and squared curvature regularization), which preserves long structures, but also introduces block artifacts. El-Zehiry and Grady are limited by 90 degree resolution, i.e. their results are composed of blocks and contain large regions of unlabeled pixels QPBO  did not label. Schoenemann et al.  obtain block artifacts with increasing regularization. Strandmark and Kahl  do not preserve fine details and also suffer from some angular resolution artifacts. In contrast, our method (neighborhood size ) preserves fine details in the segmentation and produces clean object boundaries for weak and strong regularization without requiring excessive runtimes.
|(a) original||(b)  small||(c)  large||(d)  small||(e)  large||(f) ours small||(g) ours large|
|(h) original||(i)  small||(j)  large||(k)  small||(l)  large||(m) ours small||(n) ours large|
5 Conclusion and Future Work
In this paper we proposed a novel approach to squared curvature computation and regularization. We gave an integral geometric derivation of our method which justifies that straight triple cliques can be used to measure squared curvature. Our triple cliques decompose into a set of submodular and supermodular pairwise cliques, which can be efficiently optimized by LSA-TR. We showed that our approach works for high angular resolutions and thus does not suffer from grid artifacts as do previous methods. The results demonstrate that we outperform these methods in terms of quality and efficiency.
This approach naturally extends to higher dimensions for the regularization of 3D surfaces, which will be important e.g. for 3D reconstruction approaches. We will leave this for future work.
6 Appendix: Proof of Theorem 1
We first state that the partial area of the circle of curvature above a half chord of length in Figure 13(a) is given by
Based on this expression we can now derive the area in Figure 13(b) with respect to a half chord of length in a circle of radius
Using Taylor approximation w.r.t. we obtain
-  E. Boros, P. Hammer, and X. Sun. Network flows and minimization of quadratic pseudo-boolean functions. Technical report RRR 17-1991, RUTCOR, 1991.
-  Y. Boykov and M.-P. Jolly. Interactive graph cuts for optimal boundary & region segmentation of objects in n-d images. In Int. Conf. of Computer Vision, Vancover, Canada, 2001.
-  Y. Boykov and V. Kolmogorov. Computing geodesics and minimal surfaces via graph cuts. In International Conference on Computer Vision (ICCV), 2003.
-  A. Bruckstein, A. Netravali, and T. Richardson. Epi-convergence of discrete elastica. Applicable Analysis, 79(1-2):137–171, 2001.
N. El-Zehiry and L. Grady.
Fast global optimization of curvature.
Conf. Computer Vision and Pattern Recognition, 2010.
-  L. Gorelick, Y. Boykov, O. Veksler, I. BenAyed, and A. Delong. Submodularization for Quadratic Pseudo-Boolean Optimization. In ArXiv, 2013.
-  S. Heber, R. Ranftl, and T. Pock. Approximate envelope minimization for curvature regularity. In Proceedings of the 12th international conference on Computer Vision - Volume Part III, European Conf. on Computer Vision, pages 283–292, 2012.
-  V. Kolmogorov. Convergent tree-reweighted message passing for energy minimization. IEEE Transanctions on Pattern Analysis and Machine. Intelligence., 28:1568–1583, October 2006.
Reverend bayes on inference engines: A distributed hierarchical
National Conference on Artificial Intelligence, pages 133–136, 1982.
-  C. Rother, V. Kolmogorov, V. Lempitsky, and M. Szummer. Optimizing binary mrfs via extended roof duality. In Conf. Computer Vision and Pattern Recognition, pages 1–8, 2007.
-  T. Schoenemann, F. Kahl, S. Masnou, and D. Cremers. A linear framework for region-based image segmentation and inpainting involving curvature penalization. Int. Journal of Computer Vision, 2012.
-  A. Shekhovtsov, P. Kohli, and C. Rother. Curvature prior for MRF-based segmentation and shape inpaint. In DAGM, 2012.
-  P. Strandmark and F. Kahl. Curvature regularization for curves and surfaces in a global optimization framework. In EMMCVPR, pages 205–218, 2011.