The Candidate Multi-Cut for Cell Segmentation

by   Jan Funke, et al.

Two successful approaches for the segmentation of biomedical images are (1) the selection of segment candidates from a merge-tree, and (2) the clustering of small superpixels by solving a Multi-Cut problem. In this paper, we introduce a model that unifies both approaches. Our model, the Candidate Multi-Cut (CMC), allows joint selection and clustering of segment candidates from a merge-tree. This way, we overcome the respective limitations of the individual methods: (1) the space of possible segmentations is not constrained to candidates of a merge-tree, and (2) the decision for clustering can be made on candidates larger than superpixels, using features over larger contexts. We solve the optimization problem of selecting and clustering of candidates using an integer linear program. On datasets of 2D light microscopy of cell populations and 3D electron microscopy of neurons, we show that our method generalizes well and generates more accurate segmentations than merge-tree or Multi-Cut methods alone.



There are no comments yet.


page 6


Probabilistic Evaluation of Candidates and Symptom Clustering for Multidisorder Diagnosis

This paper derives a formula for computing the conditional probability o...

Phase Contrast Microscopy Cell PopulationSegmentation: A Survey

Phase contrast microscopy (PCM) has been widely used in biomedicine rese...

Beyond Hartigan Consistency: Merge Distortion Metric for Hierarchical Clustering

Hierarchical clustering is a popular method for analyzing data which ass...

Learning to segment clustered amoeboid cells from brightfield microscopy via multi-task learning with adaptive weight selection

Detecting and segmenting individual cells from microscopy images is crit...

Candidates v.s. Noises Estimation for Large Multi-Class Classification Problem

This paper proposes a method for multi-class classification problems, wh...

Microtubule Tracking in Electron Microscopy Volumes

We present a method for microtubule tracking in electron microscopy volu...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction


Figure 1: Illustration of CMC compared to a merge-tree and Multi-Cut segmentation. In the superpixel hierarchy shown on the left, there is no candidate producing the desired segmentation of the cell. Consequently, the best merge-tree segmentation (which does not use the adjacency edges between candidates) is the selection of candidates and as two separate objects (highlighted in orange) which introduces a split error. CMC, in contrast, introduces adjacency edges between all neighboring candidates. Therefore, CMC can pick candidates and , and the adjacency edge

to merge these candidates (highlighted in blue). This way, a segment can be assembled that is not limited by the heuristic superpixel hierarchy. The Multi-Cut on the same initial superpixels could possibly achieve the same segmentation, but has to resort to smaller superpixels that lack structural information (highlighted in green).


(a) candidate mc


(b) valid selection


(c) invalid selection
Figure 5: Illustration of the proposed model. LABEL:sub@fig:cmc:overview:cmc Lower part: Given initial supervoxels (,,, and ), a heuristic region merging strategy is used to generate larger candidates (,, and ). Upper part: The candidate region adjacency graph (CRAG) is used to represent all candidates, their adjacencies across all levels (solid gray lines), and subset relations (black arrows). LABEL:sub@fig:cmc:overview:valid A selection of candidates and adjacency edges producing a valid segmentation. Note that this segmentation can not be produced by the merge-tree method alone, since it can not merge and . A Multi-Cut method could generate the same segmentation by merging , , and

, but would not be able to exploit the features extracted on the larger candidate

. LABEL:sub@fig:cmc:overview:invalid An invalid selection of candidates and adjacency edges: Candidates , overlap with and can not be selected with at the same time. Further, is merged both with and , but and are not merged, thus violating the transitivity of equivalence.

In this paper, we are addressing the problem of segmenting multiple objects in biomedical images, possibly against background. For this problem, merge-tree methods and Multi-Cut methods are amongst the best performing for a range of data modalities.

Both methods start with an initial set of superpixels, that is assumed to provide an oversegmentation. In merge-tree methods, these superpixels are iteratively merged to obtain a hierarchy of candidate segments. Amongst all candidates, a cost-minimal and non-overlapping subset is selected to yield a segmentation. The advantage of these methods is that they can consider candidates larger than the initial superpixels and thus use more meaningful features. A clear disadvantage is the limited expressiveness, as these methods require that each object is correctly segmented by one candidate in the merge-tree. Merge-tree methods demonstrate state-of-the-art performance for the segmentation of cells in 2D light microscopy [, , ]. Variations of this approach have also been successfully applied to the segmentation of neurons in volumes of electron microscopy [, ], but were ultimately outperformed by Multi-Cut methods []. Multi-Cut methods consider finding a segmentation as an instance of a clustering problem on superpixels [, ]. For that, edges in an adjacency graph of superpixels are cut. A segmentation is obtained as the connected components of a cost-minimal cut, where constraints ensure that there is no path connecting two separated superpixels. In contrast to merge-tree methods, a correct segmentation can theoretically always be obtained, if the initial superpixels are oversegmenting. However, small superpixels carry the risk of not capturing meaningful features, like the local orientation of a cellular structure or the diameter of a cell, which could help to resolve ambiguities during inference. Multi-Cut methods are the current state of the art for the segmentation of neurons in electron microscopy volumes [, , , ], but are outperformed by merge-tree methods on the segmentation of cells in light microscopy images [].

The specific advantages and disadvantages of the two methods make them perform differently depending on the characteristics of a given dataset. In particular, none of the two methods performs well on both 2D segmentation of cells in light microscopy and 3D segmentation of neurons in electron microscopy.

To combine the advantages of both methods (larger feature context of merge-tree methods and the expressiveness of Multi-Cut methods), we introduce a segmentation model that jointly selects and

clusters segment candidates from a merge tree. First, we obtain a merge-tree of segment candidates following greedy merging on initial superpixels. We then introduce adjacency edges between all adjacent candidates across all levels of the tree and train a classifier on ground-truth to obtain a cost for the selection of each candidate and each merge of an adjacency edge. Finally, we find the globally cost-minimal selection and clustering of candidates by solving an integer linear program (ILP) with a cutting-plane method. Compared to the standard formulations for merge-tree and Multi-Cut segmentations, our model has two advantages:

First, our model unifies the two method families including both as special cases (see Fig. 1). A merge-tree segmentation in the style of [, ] can be obtained by simply omitting the adjacency edges. Similarly, the Multi-Cut formulation [, ] can be obtained by omitting candidates other than the initial superpixels. In our model, however, a valid solution allows to select higher-level candidates and merge them with lower-level candidates. This allows us to train a classifier on more meaningful features that are only available for larger candidates. In contrast to merge-tree methods, however, we are not limited by the choice of the extracted candidates. Every possible segmentation given the initial superpixels can still be realized.

Second, by allowing candidates to not be selected at all, our formulation is particularly well suited to segment several foreground objects against background. This is required in 2D cell segmentation from light microscopy images where foreground objects are not tiling the plane. Our model has a dedicated cost contribution for the selection of candidates which depends on features of foreground objects. This is in contrast to previous Multi-Cut methods that required a post-processing step to filter background segments from foreground segments [, ].

We evaluate our model in two different and dissimilar setups: First, on the segmentation of cells in 2D light microscopy (involving three datasets of different resolution, microscopy modality, and cell types), and second, on the reconstruction of neurons from electron microscopy volumes of neural tissue. Our model shows a consistent improvement over both merge-tree methods and Multi-Cut methods. This is of particular interest since neither merge-tree methods nor Multi-Cut methods deliver state-of-the-art performance on both datasets jointly.

2 Method

In order to combine the advantages of merge-tree methods and Multi-Cut methods, we introduce a model generalizing both: the Candidate Multi-Cut (CMC). In our model, the standard Multi-Cut formulation [] is extended by considering additional candidate segments formed by merging initial superpixels to obtain a merge-tree. Various merging strategies can be used to obtain a merge-tree. In Section 3, we show two strategies for the datasets used here.

Given a merge-tree, we are addressing a segmentation problem in terms of the selection of candidate regions and adjacency edges (see Fig. (a)a). Let be the candidate region adjacency graph (CRAG), where is the set of all candidate regions (including the original superpixels), the set of undirected edges indicating region adjacencies across all levels of the merge-tree, the set of directed edges indicating subset relations of the candidate regions of the merge-tree, and and are cost functions for the selection of candidates and adjacency edges to merge, respectively. These costs are trained on features extracted for candidates and adjacency edges, see Section 3 for details. We encode a selection and merging of candidates with binary indicator variables and . Setting means that the candidate represented by node is part of an object (as opposed to being considered background). Setting states that the adjacent candidates and are part of the same object. By rewriting the costs and

as vectors

and , such that they are congruent to and , we find a cost-minimal segmentation by minimizing


However, not every assignment of the indicators and results in a valid segmentation. We ensure consistency with the introduction of three types of constraints: overlap constraints ensure that no overlapping candidates are selected at the same time, incidence constraints force incident candidates of selected adjacency edges to be selected as well, and path constraints state that for every adjacent pair of candidates that are not merged, there is no path of selected adjacency edges connecting them indirectly. See Fig. (c)c for an illustration of these constraints. More formally, we require:


For the overlap constraints in Eq. 2, denotes the set of all conflict cliques, i.e., a set of candidates that are mutually overlapping. For tree-shaped candidate subset graphs like those used here, the set simply contains all candidates of all paths from a leave node to the root node. In the example in Fig. (a)a, . The incidence constraints in Eq. 3 force the indicators and to be selected, if is selected. Finally, the path constraints in Eq. 4 ensure that if an adjacency edge was not selected (i.e., ), there is no path of selected adjacency edges connecting them otherwise (i.e., the sum of selected edges along the path is strictly smaller than the length of the path). denotes the set of all paths on adjacency edges connecting candidates and . Since there are in general exponentially many paths connecting two candidates in a CRAG, we do to not add these constraints initially. Following a cutting plane strategy, we solve an ILP without those constraints, and add violated constraints as needed and resolve until a consistent solution is found.

3 Experiments and Results


Figure 6: Samples of the segmentations obtained on datasets A, B, and C (from top to bottom), using the Multi-Cut formulation (MC) [], the merge-tree segmentation (MT) [], and our model (CMC).


Merge-Tree Generation.