Since the proliferation of graph-cuts as an image segmentation mechanism, there has been a great amount of interest in expanding the methods to address problems in which more global information about the segmented objects is known. Topological considerations in terms of label orderings have allowed objects to be broken into and reassembled from distinct constituents, resulting in linear (or Ishikawa) models, submodular graph models, and hierarchical models. Global knowledge about the geometry of individual labels have led to approaches such as graph-cut-based shape templates and graph-cut-accelerated statistical shape models.
Shape information that can be implemented within a single graph-cut are particularly interesting in that they maintain submodularity and therefore guarantee global optimality. Often, these are formulated as constraints on the segmentation itself via additional infinite-cost edges between nodes. Simple star convexity and geodesic star convexity in particular use infinite-cost edges to constrain objects to be connected to a predefined vantage point through either a straight path (for simple star convexity) or a geodesic (for geodesic star convexity).
For both label ordering considerations and shape information, continuous counterparts to discrete graph cuts have been developed. In terms of label ordering, Bae et al. extended the Ishikawa model. Baxter et al.[10, 11] extended the hierarchical model and developed a framework for arbitrary label orderings. For simple and geodesic star convexity, Yuan et al. and Ukwatta el al. provided two alternative solutions in the continuous domain. However, there has yet to be an general-purpose, extendable framework that combines topological and shape considerations simultaneously.
The contribution of this paper is the definition of shape complexes. Shape complexes are a collection of partially ordered labels in which several are constrained to have geodesic star convexity, thus addressing topological and shape considerations simultaneously. The result is that many additional shape constraints (such as rings) can be represented on multiple distinct objects simultaneously while maintaining a single co-ordinate system.
3 Previous Work
Yuan et al. developed the first geodesic star convexity constraint in continuous max-flow segmentation, addressing optimization problems of the form:
where is the labeling function and is the local direction of the geodesic. This was addressed through the use of convex relaxation and primal-dual optimization using the labeling and , as well as one additional field as multipliers. was used to ensure that the inequality constraint was respected, ensuring geodesic star convexity.
A very different approach was taken by Ukwatta et al. in order to implement geodesic star convexity. In his approach, the image co-ordinate system is warped, each in-plane image cross-section being represented in cylindrical co-ordinates with the center point placed somewhere in the object of interest, i.e. the interior blood pool of the femoral arterial. When expressed as such, simple star convexity can be expressed as a topological constraint, that is, as a continuous Ishikawa model. This has the additional benefit that multiple simple star convexity constraints can be placed. In the framework proposed by Ukwatta et al., both the vessel interior and the entire vessel (the union of the interior and wall labels) are constrained to have simple star convexity. Geodesic star convexity could be implemented by furthering warping of this co-ordinate system transformation.
However, these resampling approaches can be subject to ambiguity and ill-definition in a variety of cases. For the sake of example, consider vascular segmentation in which the vessel being segmentation bifurcates. After that bifurcation, no single center point can be defined, thus rendering the co-ordinate transformation ill-defined. The transformation may be ambiguous when the vascular ‘bends back’ on itself creating a U shape, causing multiple slices to intersect after warping. Illustrative versions of these three cases (including the non-ambiguous, well-defined case) are shown in Figure 1.
Ultimately, the issues associated with co-ordinate system warping become more intractable as the number of distinct labels with geodesic star convexity increases, resulting in more ill-definition or warping ambiguity, as multiple interacting co-ordinate systems must be defined.
4 Shape Complexes in Continuous Max-Flow Segmentation
4.1 Continuous Max-Flow Model
This paper extends the work of Baxter et al. on generalized hierarchical continuous max-flow (GHMF) models. Baxter et al. used the method of primal-dual optimization with augmented Lagrangian multipliers to address a max-flow min-cut problem through a directed tree of continuous spaces. The directed tree or hierarchy encodes the label ordering in the form of super-labels partitioned into their constituent parts with leaf nodes representing the ultimate partition of the whole image. The minimization equation we intend to solve extends the GHMF minimization equation:
by the addition of the constraint originally posed by Yuan et al.:
This constraint dictates that for each label at each pixel, there is a direction in which assigning the label to voxel entails giving the same label to voxels proximal to in the positive
direction. If all of these direction vectors point at a single spatial location, this enforces simple star convexity. If this direction field is warped, it enforces geodesic star convexity.
4.2 Primal Formulation
As in the previous work, the primal formulation will be expressed as a max-flow optimization, specifically:
subject to the capacity constraints:
and the flow conservation constraint:
, the spatial flow not including flow along the infinite capacity direction , is distinguished from , the overall spatial flow, noting that . Here we depart from the approach taken by Yuan et al. by rewriting Eq. (3) in terms of instead of yielding the formula:
subject to the constraints:
in which it should be noted that can be explicitly solved for, appearing only in the constraints and . In order to ensure the spatial capacity constraint, the value of should be minimized over . This can be done analytically by finding the projection of onto , ignoring any component of in the direction. Thus, assuming without loss of generality that is unit length if a geodesic direction is specified and zero length if not. The final primal formulation is:
subject to the constraints:
4.3 Primal-Dual Formulation
In order to develop the primal-dual formulation, we use the method of Lagrangian multipliers over the primal equation (4) on the flow conservation constraint , yielding the equation:
To ensure that this equation is optimizable, meeting the requirements of the minimax theorem, we must ensure that it is convex with respect to , considering to be fixed, and concave with respect to with fixed. Holding and fixed, is also fixed and thus Eq. (5) is linear with no constraints and is therefore concave. If is fixed, Eq. (5) is linear with convex constraints and is therefore convex. Thus Eq. (5) can be optimized.
4.4 Dual Formulation
To show the equivalence of Eqs. (2) and (5), we will consider the optimization of each set of flows in a bottom-up manner similar to GHMF. Starting with any leaf node in the tree we can isolate in (5) as:
(If , the function can be arbitrarily maximized by .) Every branch label can then be isolated as:
at the saddle point defined by . The source flow, , can be isolated similarly as:
at the saddle point defined by . These constraints combined yield the labeling constraints in the original formulation.
To maximize the spatial flows, we will use the first primal formulation Eq. (3) and separate the spatial flows into and . Considering first, the maximization can be expressed as described by Giusti ] as:
Taking the same approach to :
which can be arbitrarily maximized if by allowing . Thus, acts as a multiplier over the constraint and ensures the geodesic star convexity constraint on Eq. (2).
4.5 Pseudo-Flow Model
An alternative approach can be found by taking advantage of the pseudo-flow framework for GHMF explored by Baxter et al. Ultimately, the framework is equivalent with the exception of the Chambolle iteration in which the exemption flow must be removed prior to the projection operation and subsequently added back in. Thus, the spatial flow update step becomes:
5 Solution to Primal-Dual Formulation
where is an additional positive parameter penalizing deviation from the flow conservation constraint.
Eq. (12) can be optimized using each variable independently. We use the following steps iteratively:
The calculation, subtraction, and subsequent addition of allows for the descent to occur over the entire spatial flow but the spatial capacity to be applied only to the flow in directions not exempted by the infinite-capacity direction .
Maximize (12) over at the leaves analytically by:
Maximize (12) over at the branches analytically by:
Maximize (12) over analytically by:
Minimize (12) over analytically by:
5.1 Generalized Hierarchical Max-Flow Algorithm with Geodesic Star Convexity
5.1.1 Augmented Lagrangian Approach
The algorithm used is an extension of the GHMF algorithm, taking advantage of its general framework for label ordering. The key difference between algorithms is the use of and to calculate how much of the spatial flow is exempted from the spatial capacity constraint . This exempted flow is removed prior to the projection operation and added back in after said operation. The sequential algorithm used is:
which makes use of the function:
For the sake of conciseness, the ‘for do ’ loops surrounding each assignment operation have been replaced with the prefix in both the algorithm and the function definitions. The initialization mechanism, InitializeSolution(S), is flexible and the current algorithm initializes as . Alternative approaches could include the original GHMF initialization using the minimum cost solution with no regularization or shape constraints.
5.1.2 Pseudo-Flow Approach
The pseudo-flow approach can be made into the following algorithm, using the notation from Baxter et al.:
where is the set of all leaf labels (where ). and are top-down and bottom-up orderings through the hierarchy respectively.
6 Discussion and Conclusions
In this paper, we present an algorithm for addressing continuous max-flow problems in which shape complexes, collections of ordered labels with geodesic star convexity constraints, are defined. Shape complexes allow for a variety of more complex shapes to be represented, such as rings, as well as multiple distinct complexes without possibly ill-defined or ambiguous co-ordinate warping transformations.
In terms of future work, there are many different directions in which to take shape complexes. From an implementation point of view, the next steps will be the incorporation of shape complexes into more flexible label ordering models such as directed acyclic graph max-flow models which would increase the flexibility of the framework both in terms of possible models for regularization.
Acknowledgements.The authors would like to acknowledge Dr. Jiro Inoue, Dr. Elvis Chen and Jonathan McLeod for their invaluable discussion, editing, and technical support.
-  Y. Boykov, O. Veksler, and R. Zabih, “Fast approximate energy minimization via graph cuts,” Pattern Analysis and Machine Intelligence, IEEE Transactions on 23(11), pp. 1222–1239, 2001.
-  H. Ishikawa, “Exact optimization for markov random fields with convex priors,” Pattern Analysis and Machine Intelligence, IEEE Transactions on 25(10), pp. 1333–1336, 2003.
-  A. Delong and Y. Boykov, “Globally optimal segmentation of multi-region objects,” in Computer Vision, 2009 IEEE 12th International Conference on, pp. 285–292, IEEE, 2009.
-  A. Delong, L. Gorelick, O. Veksler, and Y. Boykov, “Minimizing energies with hierarchical costs,” International journal of computer vision 100(1), pp. 38–58, 2012.
N. Vu and B. Manjunath, “Shape prior segmentation of multiple objects with
graph cuts,” in
Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, pp. 1–8, IEEE, 2008.
-  J. Malcolm, Y. Rathi, and A. Tannenbaum, “Graph cut segmentation with nonlinear shape priors,” in Image Processing, 2007. ICIP 2007. IEEE International Conference on, 4, pp. IV–365, IEEE, 2007.
-  O. Veksler, “Star shape prior for graph-cut image segmentation,” in Computer Vision–ECCV 2008, pp. 454–467, Springer, 2008.
-  V. Gulshan, C. Rother, A. Criminisi, A. Blake, and A. Zisserman, “Geodesic star convexity for interactive image segmentation,” in Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pp. 3129–3136, IEEE, 2010.
-  E. Bae, J. Yuan, X.-C. Tai, and Y. Boykov, “A fast continuous max-flow approach to non-convex multi-labeling problems,” in Efficient Algorithms for Global Optimization Methods in Computer Vision, pp. 134–154, Springer, 2014.
-  J. S. Baxter, M. Rajchl, J. Yuan, and T. M. Peters, “A continuous max-flow approach to general hierarchical multi-labeling problems,” arXiv preprint arXiv:1404.0336 , 2014.
-  J. S. Baxter, M. Rajchl, J. Yuan, and T. M. Peters, “A continuous max-flow approach to multi-labeling problems under arbitrary region regularization,” arXiv preprint arXiv:1405.0892 , 2014.
-  J. Yuan, W. Qiu, E. Ukwatta, M. Rajchl, Y. Sun, and A. Fenster, “An efficient convex optimization approach to 3d prostate mri segmentation with generic star shape prior,” Prostate MR Image Segmentation Challenge, MICCAI 7512, 2012.
-  E. Ukwatta, J. Yuan, W. Qiu, M. Rajchl, B. Chiu, S. Shavakh, J. Xu, and A. Fenster, “Joint segmentation of 3d femoral lumen and outer wall surfaces from mr images,” in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2013, pp. 534–541, Springer, 2013.
-  J. Yuan, E. Bae, and X.-C. Tai, “A study on continuous max-flow and min-cut approaches,” in Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pp. 2217–2224, IEEE, 2010.
-  I. Ekeland and R. Temam, “Convex analysis and 9 variational problems,” 1976.
-  E. Giusti, Minimal surfaces and functions of bounded variation, no. 80, Springer Science & Business Media, 1984.
-  J. S. Baxter, M. Rajchl, J. Yuan, and T. M. Peters, “A proximal bregman projection approach to continuous max-flow problems using entropic distances,” arXiv preprint arXiv:1501.07844 , 2015.
-  D. P. Bertsekas, “Nonlinear programming,” 1999.
-  A. Chambolle, “An algorithm for total variation minimization and applications,” Journal of Mathematical imaging and vision 20(1-2), pp. 89–97, 2004.