Multi-region segmentation involves the partitioning of an image into multiple disjoint regions, each of which competes for space in the image. The rationale behind such a partitioning in an optimization-based manner is wide and multifaceted. In medical imaging, this approach can be used for segmentation and the identification of anatomy of interest from medical datasets. In computer vision, these approaches can be used for decomposing a scene into multiple regions[1, 2] or reconstructing images heavily polluted with noise .
Binary graph-cuts  were arguably the first instantiation of this approach, but have been necessarily limited, with no inherent extensibility. The binary model in a continuous space has been addressed by Yuan et al. , specifically, the solution to the equation:
The technique used in the continuous max-flow model made use of augmented Lagrangian multipliers applied to the convex relaxation of the above equation. The solution to the relaxed form could then be thresholded, yielding the optimal solution to the case of discrete label values.
The simplest model reflecting this partition is the Potts model.  This model has been largely explored by the computer vision community, yielding a proof of the NP-hardness of such a problem and the proliferation of methods for quickly approximating optimal solutions . Yuan et al.  have also addressed the corresponding continuous optimization problem:
However, the Potts model is limited in that it does not allow for regions to take on variable smoothness specific to their own requirements, or to collect into super-objects expressing more complex anatomy.
The first movement towards incorporating structure and variable regularization was the discrete Ishikawa model  which required the objects being segmented to be placed in a linear order. This linear order is interpreted as a sub-set relationship, that is, each object is a subset of its ‘parent’ object in the ordering, and a superset of its ‘child’ object. Bae et al.  extended the work on the continuous binary min-cut problem to the continuous Ishikawa model:
using similar variational methods but a tiered continuous graph analogous to that used by Ishikawa  in the discrete case, that is, with finite capacities on intermediate flows between labels.
The Potts and Ishikawa models allowed for immediate extensibility to multiple objects or regions, but did not extend this to a flexibility in how those objects could be arranged. Such flexibility is highly desirable in segmentation problems in which complex anatomy could be modeled a complex of simpler anatomical objects under joint regularization to encourage adjacency. This has explored in the discrete domain by Delong et al.[11, 12]. An early approach to this in the continuous domain was the limited hierarchical model proposed by Rajchl et al. for myocardial scar segmentation. This hierarchical approach allowed for the myocardium and scar to be given vastly different regularization parameters than the background, necessary due to their elongated structure, without necessitating a linear order over the objects. The Hierarchical Max-Flow (HMF) method in the continuous domain was formalized and generalized by Baxter et al. , allowing for run-time selection of hierarchies and minimizing development time in the use of hierarchical max-flow segmentation methods.
Maintaining sub-modularity for a broader selection of models has been a focus in the discrete graph cuts community due to the optimizability via graph cuts, and Delong et al. have created a framework for multi-region segmentation with more general part-whole relationships being expressible. Even more recently, the continuous max-flow approach has been further generalized in Directed Acyclic Graphical Max-Flow (DAGMF) in which any set of objects can have a regularization term and these terms simultaneously applied.  Ultimately, this family of models is the most general possible, exhausting the class of models using only the union, intersection, subset, and superset operators from set theory. (However, more models could be thought of with different constraints and considerations above simply having multiple explicit part/whole relationships.)
Focusing on continuous max-flow models, one issue with complex models is the memory requirements. In the traditional full-flow formulations [8, 10, 14, 15], each label variable and flow variable are explicitly represented in memory, leading to a large memory footprint, especially for three-dimensional datasets. The benefit of such an approach is, by the theory of Lagrangian multipliers, the constraints placed on the label variables were implicit, and the algorithm had guaranteed convergence to a feasible solution. This did however imply that intermediate steps were not necessarily feasible, requiring some correction to the labels to be applied in post-processing, often lost in the process of discretization.
The primary contribution of this paper is to consolidate the theory of Bregman proximal projections and entropic distance metrics to continuous max-flow segmentation with multiple regions. This will first contain an algorithm and corresponding proof of correctness for the Potts model (with variable smoothness parameters). This will then be extended to the Ishikawa, HMF, and DAGMF models demonstrating the general applicability of this method to max-flow segmentation under any label ordering structure.
3 Convex relaxed hierarchical models and Previous Work
3.1 Previous Work
Traditionally, continuous max-flow segmentation models have involved the explicit representation of every flow variable in memory, leading to larger memory consumption. Recently, this has been addressed for binary models by Bae et al.  in two stages. The first implicitly represents the source flow noting that under the flow conservation constraint, it could be expressed analytically in terms of the sink flow and divergence of the spatial flow. The second method, which we will be extending in this paper, was called the pseudo-flow representation and implicitly represented both source and sink flows, requiring only the spatial flows and labels to be stored. In this case, the six buffers required for the full-flow version is reduced to only four buffers, a significant improvement in efficiency of memory use.
3.2 Pseudo-Flow Approach to the Potts Model
We will proceed in developing a pseudo-flow model following the approach taken by Bae et al.  starting with the continuous Potts model.
Yuan et al. have shown the equivalence between the convex relaxed continuous Potts model:
to the primal max-flow model:
through the primal dual model:
noting that in both the primal and the primal-dual formulations, the labeling constraints and become implicit.
Firstly, we need to show that the formulation can be addressed by developing the appropriate non-smooth pseudo-flow model:
which can be derived from equation (3) as shown in the following steps:
Now that a pseudo-flow representation is developed, we can take advantage of Bregman proximal projections
to optimize this formula. Consider the distance between vector-valued labeling functionsand as:
which can be verified to be a Bregman distance (when ) using the entropy function:
If we consider a feasible labeling, , we can find another proximal labeling, , which has a lower energy by addressing the optimization:
where is a positive constant. Using a Lagrangian multiplier on the constraint , we can solve for analytically as:
noting that this answer fulfills the constraint provided the same holds for . By letting the distance weighting parameter approach 0, if . Using that fact,
illustrating that the Bregman proximal method acts just as well as a smoothed version of the non-smooth pseudo-flow representation, equation (4).
By taking the gradient of equation (6) with respect to (with substituted by equation (15)) one can derive the appropriate Chambolle iteration scheme for maximizing (6) with respect to the spatial flow variables :
where is a positive gradient descent parameter.
This yields the algorithm for optimizing the continuous Potts model using repeated Bregman proximal projections and Chambolle iterations:
3.3 Pseudo-Flow Approach to Directed Acyclic Graphical Max-Flow
For the sake of generalization, we shall begin developing a pseudo-flow model for DAGMF first, applying it to the special cases of HMF and Ishikawa models. As with the Potts model, note that Baxter et al.  proved the equivalence of the DAGMF model:
to the primal model:
through the primal-dual model:
As with before, we will develop a non-smooth pseudo-flow representation from the primal-dual representation (11). To do so, we will develop some notation, specifically:
where represents the flow excess of an end-label, , taking into account the spatial flows for super-objects containing . where represents the amount of weight assigned to a super-object based on a single component, the end-label . These definitions mirror the top-down process of flow excess accumulation and the bottom-up process of label accumulation in the original DAGMF algorithm and will play such a role in the pseudo-flow formulation. gives a convenient notation for end-labels.
As with the previous method, we will develop a pseudo-flow representation which is amenable to solving through a mixture of Bregman proximal projections and Chambolle iterations. The entropic distance metric used in this formulation is:
which can be verified to be a Bregman distance (when ) using the entropy function:
This may at first not look like a true distance, since if there were no constraints on the values of where it would only be a pseudo-metric. However, it must be noted that since these are constrained, any two labelings in which the values are equal for end-labels must also have equal values for non-end-labels. Since only if and have the same values for end-labels, also implies they have the sames for non-end-labels and thus is a valid distance.
Consider the non-smooth pseudo-flow formulation:
which can be derived from equation (11) as follows:
|(which is guaranteed by the equivalence of the DAGMF formulation and the primal-dual|
|(which is guaranteed by the equivalence of the DAGMF formulation and the primal-dual|
As with earlier, we can combine this representation with the entropic distance to yield an improved labeling proximal to via:
This is equivalent to that used for the Potts model (6). Therefore we can solve for for all end-labels :
and propagate these labels upwards to get the values of where using the labeling constraints. A corollary to this is that this label update equation approaches the pseudo-flow representation as .
Lastly, we must consider updating the spatial flows, once again by finding the gradient of equation (14) with respect to . Doing so yields the update equation with positive gradient descent parameter :
All together, this framework yields the following algorithm:
where is the top-down topological ordering starting with and ending with the labels in and is the inverse thereof. Note that the labels for intermediate nodes have also become implicit, further minimizing memory load. Note that this algorithm uses the same buffer for holding the flow excess propogating top-down and the spatial flow fields for updating the spatial flow buffers propagating up, further reducing the number of buffers required.
This algorithm requires 5 buffers per end-label, 4 buffers per non-end-label not including the source , and 1 additional buffer for accumulation purposes. This is a large improvement over the full-flow approach by Baxter et al. which requires 6 buffers per end-label, 7 buffers per non-end-label not including the source , and 2 additional buffer related to the source node.
3.4 Application of Pseudo-Flow to Ishikawa and other Hierarchical Models
It should be noted that the HMF and Ishikawa models are special cases of the DAGMF model, meaning that the same algorithm could be immediately applied to them. In the case of HMF, the DAGMF algorithm is equivalent to the algorithm yielded by repeating the process of developing a pseudo-flow representation and solving it using the aforementioned steps. In this case, it may be somewhat simplified as the orderings and correspond to simple bottom-up and top-down tree traversals, which can be readily implemented using recursion.
The Ishikawa model is more interesting in that it’s formulation renders many of the flow variables redundant, and a separate instantiation of the pseudo-flow algorithm would be warranted. Modifications to the DAGMF pseudo-flow algorithm for an Ishikawa model with levels and therefore labels, yield:
This algorithm involves buffers for the labelings, for an accumulator, for flow excess values (noting that is largely a mathematical formality and always takes on a value of during propagation or during label updates) and for spatial flows. In total, this yields buffers which is an improvement over the buffers required to efficiently implement the algorithm Bae et al. 
4 Discussion and Conclusions
In this paper, we have presented a framework for the incorporation of entropic distance and Bregman proximal projections into the optimization of the Potts model. This approach can be extended to the case of Ishikawa models, hierarchical models, and general set-theoretic models with minimal additional theoretical development, demonstrating its general applicability in continuous max-flow segmentation.
The pseudo-flow approach has the theoretical advantage over the full-flow model in that it constrains intermediate segmentations to be feasible allowing for meaningful truncation. It has the practical advantage of requiring a significantly smaller memory footprint that makes it more amenable to implementation on massively parallel computing graphics hardware where memory is available in as large a quantity.
Acknowledgements.The authors would like to acknowledge Dr. Elvis Chen and Jonathan McLeod for their invaluable discussion, editing, and technical support.
-  J.-M. Beaulieu and M. Goldberg, “Hierarchy in picture segmentation: A stepwise optimization approach,” Pattern Analysis and Machine Intelligence, IEEE Transactions on 11(2), p. 150–163, 1989.
-  A. Delong and Y. Boykov, “Globally optimal segmentation of multi-region objects,” in Computer Vision, 2009 IEEE 12th International Conference on, pp. 285–292, IEEE, 2009.
-  E. Bae, J. Yuan, X.-C. Tai, and Y. Boykov, “A fast continuous max-flow approach to non-convex multilabeling problems,” 2011.
-  Y. Y. Boykov and M.-P. Jolly, “Interactive graph cuts for optimal boundary & region segmentation of objects in ND images,” in Computer Vision, 2001. ICCV 2001. Proceedings. Eighth IEEE International Conference on, 1, p. 105–112, 2001.
J. Yuan, E. Bae, and X.-C. Tai, “A study on continuous max-flow and min-cut
Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, p. 2217–2224, 2010.
-  R. B. Potts, “Some generalized order-disorder transformations,” in Proceedings of the Cambridge Philosophical Society, 48, p. 106–109, 1952.
-  Y. Boykov, O. Veksler, and R. Zabih, “Fast approximate energy minimization via graph cuts,” IEEE Transactions on Pattern Analysis and Machine Intelligence 23(11), pp. 1222–1239, 2001.
-  J. Yuan, E. Bae, X.-C. Tai, and Y. Boykov, “A continuous max-flow approach to potts model,” in Computer Vision – ECCV 2010, K. Daniilidis, P. Maragos, and N. Paragios, eds., Lecture Notes in Computer Science, pp. 379–392, Springer Berlin Heidelberg, Jan. 2010.
-  H. Ishikawa, “Exact optimization for markov random fields with convex priors,” IEEE Transactions on Pattern Analysis and Machine Intelligence 25(10), pp. 1333–1336, 2003.
-  E. Bae, J. Yuan, and X.-C. Tai, “Global minimization for continuous multiphase partitioning problems using a dual approach,” International journal of computer vision 92(1), p. 112–129, 2011.
-  A. Delong, L. Gorelick, F. R. Schmidt, O. Veksler, and Y. Boykov, “Interactive segmentation with super-labels,” in Energy Minimization Methods in Computer Vision and Pattern Recognition, pp. 147–162, Springer, 2011.
-  A. Delong, L. Gorelick, O. Veksler, and Y. Boykov, “Minimizing energies with hierarchical costs,” International journal of computer vision 100(1), pp. 38–58, 2012.
-  M. Rajchl, J. Yuan, J. White, E. Ukwatta, J. Stirrat, C. Nambakhsh, F. Li, and T. Peters, “Interactive hierarchical max-flow segmentation of scar tissue from late-enhancement cardiac MR images,” IEEE Transactions on Medical Imaging , 2014.
-  J. S. Baxter, M. Rajchl, J. Yuan, and T. M. Peters, “A continuous max-flow approach to general hierarchical multi-labeling problems,” arXiv preprint arXiv:1404.0336 , 2014.
-  J. S. Baxter, M. Rajchl, J. Yuan, and T. M. Peters, “A continuous max-flow approach to multi-labeling problems under arbitrary region regularization,” arXiv preprint arXiv:1405.0892 , 2014.
-  L. M. Bregman, “The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming,” USSR computational mathematics and mathematical physics 7(3), pp. 200–217, 1967.
-  E. Bae, X.-C. Tai, and J. Yuan, “Maximizing flows with message-passing: Computing spatially continuous min-cuts,” in Energy Minimization Methods in Computer Vision and Pattern Recognition, pp. 15–28, Springer, 2015.
-  A. Chambolle, “An algorithm for total variation minimization and applications,” Journal of Mathematical imaging and vision 20(1-2), p. 89–97, 2004.