Image segmentation is a challenging task for which often times the use of suitable prior knowledge about the shape of the sought objects plays an important role. One interesting shape prior is convexity [9, 10, 7, 6]. In natural images, it often occurs that there are multiple convex structures of the same or different classes present in one image, such as wheels or various fruit, as well as composite structures constructed from convex parts, such as bricks and floor tiles. Biology similarly gives numerous examples of multiple convex structures. For example, many cell types are convex, such as bacteria, yeast, and more complicated cells densely packed into tissue. Numerous subcellular structures, including nuclei and various types of vesicles, are also convex. Despite the clear relevance of this situation to the task of image segmentation, respective priors have, to the best of our knowledge, not yet been addressed in the literature.
Existing methods for segmenting multiple convex structures are specifically designed for certain shapes like ellipsoids or rods, as e.g. . Such methods do not enforce generic convexity, but instead employ priors of specific shapes that happen to be convex. Furthermore, such methods commonly segment multiple structures sequentially, and neither the reconstruction of individual structures nor the resulting segmentation of multiple structures is globally optimal w.r.t. the underlying objective.
Beyond specific shape priors, there has been recent interest in generic convexity priors for binary image segmentation. Star convexity priors were introduced in , where convexity is defined with respect to all rays emanating from a central, user-defined seed point. This approach was generalized to the case of Geodesic Star Convexity by , which defines convexity with regards to Geodesic paths. Truly convex objects were first handled by . However, this approach is limited to a single foreground class, which must be explicitly modeled. The task of segmenting convex foreground objects without the requirement of user input or explicit modeling is studied in . They propose a graphical model with triple-cliques that encode convexity constraints as 1-0-1 label sequences along straight lines in the image. This formulation captures the global nature of convexity. However, it implies that only one connected component of the foreground class can be present. Furthermore, the complexity of the problem requires the use of approximate solvers which may lead to local minima. None of the above methods is able to segment many generic convex objects of multiple foreground classes, fully automatically, without the need for user-defined seed points.
Contributions. In this work we propose the first model and solver for pixel-level segmentation of many generic convex objects of multiple foreground classes. We introduce new models that include convexity constraints into multicut problems, which are ILP formulations of image decomposition problems. We consider two multicut problems, one equivalent to the correlation clustering problem , another one equivalent to the Potts model. For efficiency, following the idea in [8, 1], we iteratively incorporate only the constraints that are violated per instance, and when no more violations occur, we are guaranteed the globally optimal solution. To the best of our knowledge, our models are the first to handle many convex objects and multiple foreground classes. Our models can be solved to global optimality in small yet practical cases. Figure 1 gives an overview of the multicut problems as well as our proposed convexity constraints, as described in detail in Sections 2 and 3.
2 Image Decomposition by Multicuts
Given the pixel grid graph of an image, image segmentation tasks are often modeled as the problem of assigning, to each node , one label from a label set so as to minimize some objective function. A widely used objective function is the energy of a pairwise conditional random field. A particular instance of this type of objective is the well-known Potts model. Its pairwise terms are non-zero only for edges that connect nodes with different labels. A special case of the Potts model is the Correlation Clustering or Partitioning model . The respective energy neglects unary terms, and the number of labels equals the number of nodes.
In the following we describe two Multicut Problems. The first one is equivalent to Correlation Clustering. The second one is equivalent to the Potts model. We also discuss respective solvers as proposed in [8, 1]. These Multicut Problems are ILPs that form the basis of our contribution: In Section 3 we take the set theoretic definition of convexity of the connected components of a segmentation, and directly translate it into inequality constraints that we add to the respective ILPs.
The Minimum Cost Multicut Problem.
The minimum cost multicut problem  is equivalent to the correlation clustering problem. Its feasible solutions are all decompositions of the graph . A decomposition of is a partition of such that, for every , the subgraph of induced by is connected. A subgraph of that is connected and induced by a node set is called a component of . Any decomposition of is characterized by the subset of edges that straddle distinct components: . Such a subset of edges is called a multicut of . There is exactly one multicut related to each decomposition of . The following Theorem forms the basis of multicut problems:
The multicuts of are precisely the subsets for which every cycle in satisfies .
Let be a 01-encoding of a multicut . It makes explicit, for any pair of neighboring nodes, whether and are in distinct components, namely iff .111Note that in  the interpretation of the 01-encoding is flipped, i.e. means that an edge is an element of the multicut. These edge labels allow for an equivalent formulation of as linear inequality constraints (2), which leads to the following Definition:
 Given a finite, simple, non-empty graph and a map (that is, for any pair of neighboring nodes, a cost or reward for and being in distinct components), the instance of the Minimum Cost Multicut Problem (MC) with respect to and is the ILP
The Minimum Cost Multicut Problem with Node Labels.
We also consider a Multicut Problem that is equivalent to the Potts model. This is a more constrained optimization problem in which every node assumes precisely one out of finitely many labels, and neighboring nodes are in the same component iff they have the same label:
Given a finite, simple, non-empty graph , a map , a finite set and a map (that is, for any node and any label , a cost or reward for being labeled ), the instance of the Minimum Cost Multicut Problem with Node Labels (MCN) with respect to , , and is the ILP
Here, any feasible solution is constrained such that every node is assigned at least and at most one label, namely the unique such that , by (4) and (5). It is also constrained such that, for any edge , if and only if and have distinct labels, by (6) and (7). Thus,
is the characteristic function of a multicut of. It defines uniquely a decomposition of .
Branch-and-cut algorithms for Problem 1 are proposed in [8, 1]. They find globally optimal solutions in reasonable run-time in many practical cases by including constraints (2) per instance of the problem only in case they are violated. Problem 2 is solved by  by transforming it into an equivalent Minimum Cost Multicut Problem on a modified graph. This transformation involves flipping the meaning of node label variables resulting in the so-called multiway cut problem.
This Section provides the methodological contribution of our work. In Section 3.1, we propose a model for image decomposition under the constraint that each component of the resulting partition has to be convex. We achieve this via additional inequality constraints that we include into the Minimum Cost Multicut Problem 1. Furthermore, we propose a respective model with node labels. With this model, we can enforce for any pair of labels, , that a component of label does not contain any node labeled in its convex hull. This is more general than “simply” enforcing convexity of components. We achieve this, again, via inequality constraints that we include into the Minimum Cost Multicut Problem with Node Labels 2. In Section 3.2 we propose a solver for the above optimization problems.
3.1 Convexity Constraints for Image Decomposition
Let denote an arbitrary open path in . All components of an image decomposition are convex iff for any path that does not contain any edges in , the straight line between the end points of also does not contain any edges in . We discretize this set theoretic definition of convexity as follows: In a pixel grid graph with the usual embedding into the 2d plane, where pixels are Voronoi regions of graph nodes (cf. Fig. 1a), a component is discrete convex iff for every path that does not contain any edges in , the interior of the loop formed by and the straight line between its end points does not enclose any nodes of a distinct component. See Figure 2 for a sketch. We call the set of nodes enclosed by this loop the hull of . Let denote the path in that runs along the boundary of the discrete hull of and connects the first and the last node covered by (cf. Fig. 2). All components of an image partition are discrete convex iff
We propose to formulate (8) as linear inequality constraints, which enables us to formulate the task of finding the optimal decomposition of into convex components as an ILP:
Given a finite, simple, non-empty graph and a map , the instance of the Minimum Cost Convex Component Multicut Problem (Convex-MC) with respect to and is the ILP
We also refer to this model as correlation clustering with convexity constraints, to put it into well-known terminology.
A proof of Lemma 1 is given in the Appendix.
Now we consider image decomposition with node labels, which is equivalent to the Potts model (cf. Section 2). Let denote the nodes covered by a path , and the nodes covered by the respective straight line but not by (i.e., all but the “end points” of ). Given an image decomposition with node labels , all components assigned to label are convex iff
More generally, components assigned to label have in their convex hulls only nodes with labels from a subset of (where ) iff
This holds because (5) entails . Constraints (11) are a special case of (12), namely with . We propose to formulate (12) as linear inequality constraints and hence yield an ILP that models image decomposition, with node labels, into convex components:
Given a finite, simple, non-empty graph , a map , a finite set , a map , and a set for each with the instance of the Minimum Cost Convex Component Multicut Problem with Node Labels (Convex-MCN) with respect to , , , , and is the ILP
Similar to [8, 1] we pursue a cutting plane approach in which violated cycle- and convexity constraints are separated and added incrementally. Our algorithm starts by solving MC (see Def. 1) or MCN (see Def. 2) with the method described in . Given the resulting image decomposition, we solve the Separation Problem w.r.t. convexity constraints as described below. However, solving an ILP with added convexity constraints can entail new violations of cycle constraints. Hence we propose the algorithm defined in Figure 3. Upon termination, the algorithm gives the globally optimal feasible solution. In practice we can, optionally, allow for ILPs to be solved up to some relative gap of . In this case our algorithm gives a feasible solution whose energy is at most away from the global optimum, where is the gap obtained in the last iteration, i.e. directly before termination.
Separation of Convexity Constraints. Given an image decomposition, assuming a 2D pixel grid, we detect violated convexity constraints by scanning along sequences of nodes, , each defined by a start node and an offset , with co-prime. In case of the Convex-MC problem 3, convexity is violated iff two nodes along a sequence lie in the same component, and there exists a node in the sequence with such that lies in a different component. In case of the Convex-MCN problem 4, there is a violation of (generalized) convexity iff lie in the same component, labeled , and there exists a node with such that is labeled .
The cardinality of the set of orientations of sequences, , is bounded by , where denotes the number of pixels in a 2d image. Scanning for violations along a particular sequence can be done greedily. For each orientation, the union of the sets of sequences to consider covers the whole image. Hence the computational complexity for processing one particular orientation is in . Overall, the computational complexity of scanning for violations is bounded by . Optionally, one may want to consider only a subset of orientations, . Then the effort reduces to .
Given a pair of nodes that constitutes a violation of convexity, we need to find a respective path and straight line . As for a path from to through their component: In general there are many such paths. To form a constraint, we pick just one such path. We explore two variants: In case of Convex-MC (see Def. 3), a shortest path in terms of number of edges; In case of Convex-MCN (see Def. 4), a cheapest path in terms of unaries of the label assigned to the component. Dijkstra’s algorithm yields an optimal path for either variant.
We compute the straight line as follows: (1) Determine the orientation of the loop formed by path P and the (continuous) straight line . (2) Infer the direction of the normal on the (continuous) straight line that points to the interior of this loop. Without loss of generality, we assume that and the continuous straight line do not intersect. (3) Greedily find the discrete shortest path with minimal distance of nodes to the continuous straight line, subject to the constraint that no node lies outside the loop.
4 Results and Discussion
We present proof-of-concept results of our method on various photographs and biological images. For each exemplary image, in addition to the result of our method, we also show the respective result obtained without convexity constraints. Finally, we provide a comparison to state of the art  on two exemplary images. We employ a four-connected grid graph in all experiments. If not noted otherwise, we check violations of convexity in 8 discrete directions, and set the stopping criterion for the ILP solver to 2%. This is not a hard stopping criterion, and hence smaller gaps are achieved per instance. Table 1 lists the gaps, number of iterations, run-times, and energies obtained for each of the examples.
|Nuclei, 2 label Potts||48x48||725||770||9||15 sec||0.05 %|
|Fly wing, 2 label Potts||101x101||1089||1240||11||380 sec||2 %|
|Fried eggs, 3 label Potts||163x137||64633||64567||22||166 sec||0.04 %|
|Synthetic, 3 label Potts||155x104||3275||3529||1||2 sec||0 %|
|Apple and orange, 3 label Potts||200x165||40390||40428||158||870 sec||0.06 %|
|Stone wall, correlation clustering||101x101||-226||-188||33||10519 sec||1.5 %|
|Fly wing, correlation clustering||101x101||-419||-394||14||251 sec||1 %|
Potts Model. Figure 4 shows examples to which we apply two-label Potts models enriched by convexity constraints on the foreground label by means of the Convex-MCN problem 4. Figure 4(a) shows a biological image of ellipsoidal cell nuclei in the tissue of a nematode worm. The image shows four whole nuclei, and a fraction of a fifth one at the bottom. Convexity constraints allow for a perfect segmentation of the nuclei. In contrast, a Potts model without convexity constraints fails to split apart the nuclei, and yields holes within components. Figure 4(d) shows a biological image of polygonal cells in a fly wing. Again, convexity constraints allow for a perfect segmentation of the nuclei, while a respective Potts model without convexity constraints is not able to correctly split apart the cells. Figure 5(a) shows a photograph to which we apply a three-label Potts model with convexity constraints on one foreground label. Our method is able to accurately segment two convex foreground objects on top of a second, non-convex foreground label.
Results in Figures 4 and 5(c) were achieved with “simple” convexity constraints as captured by (11). An example for the more general constraints (12) is given in Figure 5(d-f). Here, we enforce components labeled “apple” to be convex as such, while we enforce components labeled “orange” to be convex only w.r.t. the background. In other words, we allow concavities in orange-labeled components as long as they are filled exclusively by apple-labeled nodes. Consequently such concavities appear as desired. However, incorrect spurious apple components also appear within orange components. This result is interesting despite the fact that our model does not achieve a perfect segmentation, because it shows potential implications of the generalized constraints.
Correlation Clustering. Figure 6 shows two exemplary results for correlation clustering enriched by convexity constraints by means of the Convex-MC problem 3. In both examples, densely packed objects, namely bricks in a stone wall and cells in a fly wing, are nicely separated due to convexity constraints. At the same time, the space between these objects is tesselated into convex components. Although these results might not be of direct use as segmentations of the respective objects, they may well serve as convex supervoxelizations to be used as input for further processing.
Comparison to State-Of-The-Art on Two Exemplary Images.
We compare our method to the state-of-the-art for segmentation with convexity constraints . The method of Gorelick et al.  is able to handle only one convex structure of one foreground label, and we chose exemplary images accordingly (Figure 7). We use the code provided by the authors. First we study a synthetic image (Figure 7 top). The method of  initializes via the Graph Cuts solution, yielding a hole in the foreground object. In the process of resolving this high energy configuration, the method breaks the outer boundary of the object and settles to a sub-optimal solution. In contrast, our method is able to obtain the globally optimal solution. On a second, biological image (Figure 7 bottom), we ran  with two different initializations, namely the standard Graph Cut solution, as well as a box that we manually placed around the sought structure. Both initializations result in sub-optimal solutions, whereas our method is again able to obtain a globally optimal solution. However, we note that the method of  is considerably faster than ours.
5 Conclusion and Future Work
We proposed a new approach that introduces convexity constraints into two multicut problems that are equivalent to Correlation Clustering and the Potts Model, respectively. Our approach handles convexity constraints for many connected components of multiple different classes, and additionally for pre-specified convexity relationships between objects of different classes. All concepts described in this paper extend in a straightforward way to 3D. In future work we will explore strategies for improving the run-time, e.g. via warm-start of subsequent ILPs, for the application to larger data.
Proof of Theorem 1.
First we show that every multicut satisfies for all cycles. Given a decomposition and related multicut , assume there exists a cycle with . Then straddles distinct components because . On the other hand, the path connects and with edges that are not in , and hence and belong to the same component. By contradiction, this proves that for all cycles. Second we show that every that satisfies for all cycles is a multicut of . Given such a , we construct a decomposition and related multicut of as follows: there exists a path between and such that . Now we show that . If , then , because there is a path between and such that , namely . If , assume . Then, because , there exists a path between and such that . Hence the cycle satisfies . By contradicting , this proves that if , then .
Proof of Lemma 1.
-  B. Andres, J. Kappes, T. Beier, U. Köthe, and F. Hamprecht. Probabilistic image segmentation with closedness constraints. In Computer Vision (ICCV), 2011 IEEE International Conference on, pages 2611–2618, Nov 2011.
-  N. Bansal, A. Blum, and S. Chawla. Correlation clustering. Machine Learning, 56(1-3):89–113, 2004.
-  J. E. Bresenham. Algorithm for computer control of a digital plotter. IBM Systems journal, 4(1):25–30, 1965.
-  S. Chopra and M. R. Rao. The partition problem. Mathematical Programming, 59(1-3):87–115, 1993.
-  R. Delgado-Gonzalo, P. Thevenaz, C. S. Seelamantula, and M. Unser. Snakes with an ellipse-reproducing property. Image Processing, IEEE Transactions on, 21(3):1258–1271, 2012.
-  L. Gorelick, O. Veksler, Y. Boykov, and C. Nieuwenhuis. Convexity shape prior for segmentation. In D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, editors, Computer Vision – ECCV 2014, volume 8693 of Lecture Notes in Computer Science, pages 675–690. Springer International Publishing, 2014.
V. Gulshan, C. Rother, A. Criminisi, A. Blake, and A. Zisserman.
Geodesic star convexity for interactive image segmentation.
Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pages 3129–3136, June 2010.
-  J. Kappes, M. Speth, B. Andres, G. Reinelt, and C. Schnörr. Globally optimal image partitioning by multicuts. In Y. Boykov, F. Kahl, V. Lempitsky, and F. Schmidt, editors, Energy Minimization Methods in Computer Vision and Pattern Recognition, volume 6819 of Lecture Notes in Computer Science, pages 31–44. Springer Berlin Heidelberg, 2011.
-  E. Strekalovskiy and D. Cremers. Generalized ordering constraints for multilabel optimization. International Conference on Computer Vision (ICCV), 2011.
-  O. Veksler. Star Shape Prior for Graph-Cut Image Segmentation. In Computer Vision–ECCV 2008, pages 454–467. Springer Berlin Heidelberg, Berlin, Heidelberg, 2008.