Log In Sign Up

Leveraging Domain Knowledge to improve EM image segmentation with Lifted Multicuts

by   Constantin Pape, et al.

The throughput of electron microscopes has increased significantly in recent years, enabling detailed analysis of cell morphology and ultrastructure. Analysis of neural circuits at single-synapse resolution remains the flagship target of this technique, but applications to cell and developmental biology are also starting to emerge at scale. The amount of data acquired in such studies makes manual instance segmentation, a fundamental step in many analysis pipelines, impossible. While automatic segmentation approaches have improved significantly thanks to the adoption of convolutional neural networks, their accuracy still lags behind human annotations and requires additional manual proof-reading. A major hindrance to further improvements is the limited field of view of the segmentation networks preventing them from exploiting the expected cell morphology or other prior biological knowledge which humans use to inform their segmentation decisions. In this contribution, we show how such domain-specific information can be leveraged by expressing it as long-range interactions in a graph partitioning problem known as the lifted multicut problem. Using this formulation, we demonstrate significant improvement in segmentation accuracy for three challenging EM segmentation problems from neuroscience and cell biology.


page 1

page 2

page 7

page 12


FusionNet: A deep fully residual convolutional neural network for image segmentation in connectomics

Electron microscopic connectomics is an ambitious research direction wit...

Human Perception-based Evaluation Criterion for Ultra-high Resolution Cell Membrane Segmentation

Computer vision technology is widely used in biological and medical data...

Attention-Based Transformers for Instance Segmentation of Cells in Microstructures

Detecting and segmenting object instances is a common task in biomedical...

Biologically-Constrained Graphs for Global Connectomics Reconstruction

Most current state-of-the-art connectome reconstruction pipelines have t...

Multiclass Yeast Segmentation in Microstructured Environments with Deep Learning

Cell segmentation is a major bottleneck in extracting quantitative singl...

Anisotropic EM Segmentation by 3D Affinity Learning and Agglomeration

The field of connectomics has recently produced neuron wiring diagrams f...

Toward Streaming Synapse Detection with Compositional ConvNets

Connectomics is an emerging field in neuroscience that aims to reconstru...

1 Introduction

Large-scale electron microscopy (EM) imaging is becoming an increasingly important tool in different fields of biology. The technique was pioneered by efforts to map the neural circuitry of small animals at synaptic resolution to obtain their so-called connectome. In the 1980’s White et al. [37] have mapped the complete connectome of C. elegans in a manual tracing effort which spanned over a decade. Since then, the throughput of EM imaging has increased by several orders of magnitude thanks to innovations like multibeam serial section EM [10], hot-knife stitching [13] or gas cluster milling [13]. This allows to image much larger volumes up to the complete brain of the fruit-fly larva [11] and even the adult fruit-fly [41]. Recently, studies based on large-scale EM have become more common in other fields of biology as well [27, 29, 33].

Given the enormous amount of data generated, automated analysis of the acquired images is crucial; one of the key steps being instance segmentation of cells, neurons or cellular organelles. In recent years, the accuracy of automated segmentation algorithms has increased significantly thanks to the rise of deep learning-based tools in computer vision and the development of convolutional neural networks (CNNs) for semantic and instance segmentation

[3, 12, 22, 16]. Still, it is not yet good enough to completely forego human proof-reading. Out of all EM image analysis problems, neuron segmentation turned out to be particularly difficult[16] due to the small diameter and long reach of neurons and astrocytes, but other EM segmentation problems have not yet been fully automated either. One of the major downsides of CNN-based segmentation approaches lies in their limited field of view which makes them overly reliant on local boundary evidence. Heavy metal staining used in the EM sample preparation stains all cells indiscriminately and forces segmentation algorithms to rely on cell membrane detection to separate membrane-bound objects. Staining artifacts or alignment issues can severely weaken this evidence and often cause false merge errors where separate objects get merged into one. On the other hand, membranes of cellular organelles or objects of small diameter often cause false split errors where a single structure gets split into several segmented objects.

Human experts avoid many of these errors by exploiting additional prior knowledge on the expected object shape or constraints from higher-level biology. Following this observation, several algorithms have recently been introduced to enable detection of morphological errors in segmented objects [31, 42, 9]. By looking at complete objects rather than a handful of pixels, these algorithms can significantly improve the accuracy of the initial segmentation. In addition to purely morphological criteria, Krasowski et al. in [18] suggested an algorithm to exploit biological priors such as an incompatible mix of ultrastructure elements.

Building on such prior work, this contribution introduces a general approach to leverage domain-specific knowledge for the improvement of segmentation accuracy. It allows to incorporate a large variety of rules, explicit or learned from data, which can be expressed as the likelihood of certain areas in the image to belong to the same object in the segmentation. The areas can be sparse and/or spatially distant. In more detail, we formulate the segmentation problem with such rules as a graph partitioning problem with long-range attractive or repulsive edges.

Figure 1: Mapping domain knowledge to sparse lifted edges for mammalian cortex (left), drosophila brain (middle) and sponge choanocyte chamber (right). From raw data (a) to local graph problem (b) with attractive (green) and repulsive (red) edges. Domain knowledge is mapped to nodes (c) resulting in attractive (green) and repulsive (red) lifted edges (d) to obtain a final segmentation (e). For the sake of clarity, we show only a subset of lifted edges.

Horvnakova et al. in [15] showed that this problem, which they term Lifted Multicut as it corresponds to the Multicut partitioning problem with additional lifted (long-range) edges, can be solved in reasonable time for small problem sizes, while Beier et al. in [4] introduced an efficient approximate solver. Lifted Multicut with dense, short-range lifted edges has already been applied to the neuron segmentation problem [3]. There, the weights of the lifted edges were learned directly from the groundtruth segmentations. While the introduction of dense, short-range lifted edges already improves segmentation in comparison to regular Multicut partitioning, it does not allow to express explicit or learned biological knowledge on the complex object morphology or on the relations between different semantic entities. Exploiting such knowledge is the target of this contribution.

For the problem of image segmentation, the graph in the Multicut partitioning problem corresponds to the region adjacency graph of the image pixels or superpixels. The nodes of the graph can be mapped directly to spatial locations in the image. Domain knowledge expressed as rules that certain locations must or must not belong to the same object can thus be distilled into lifted edges between these locations. The weights of such lifted edges are derived from the strictness of the rules, which can colloquially range from ”usually do / do not belong to the same object” to ”always / never ever belong to the same object”.

We demonstrate the validity of this approach for three applications, incorporating different kinds of domain knowledge into the lifted multicut framework:

  • using indicators of axon/dendrite attribution to avoid merges between axonal and dendritic neural processes,

  • using learned object morphology to detect and correct falsely merged neural processes and

  • using semantic segmentation of biological structures to decrease the number of false splits in instance segmentation of sponge choanocytes.

Figure 1 gives an overview of the three applications. For all three, additional domain knowledge allows us to significantly improve segmentation accuracy. Aiming to apply the method to data of biologically relevant size, we addtionally introduce a new scalable solver for the lifted multicut problem based on our prior work from [30].

2 Related Work

Instance segmentation problems that arise in EM image analysis can be roughly separated into two categories: segmenting structures with background that touch rarely, such as synapses [20, 19, 14] or mitochondria[40] and segmenting large structures with little or no background, like neural processes or cells. For the first set of problems semantic segmentation or detection based approaches are most common, while the second is usually tackled with boundary-based instance segmentation. Here, we focus on the second use-case, but show that we can also bridge both categories in subsection 4.3.

Neuron segmentation of connectomics has been the main driver of the recent advances in boundary-based EM segmentation. Most methods ([1, 3, 28, 12, 22]) follow a three step procedure: in the first step they segment boundaries, then compute a superpixel over-segmentation and finally combine the superpixels into objects. The success of a CNN [7] in the ISBI 2012 neuron segmentation challenge [2] has prompted the adoption of this technique for the boundary prediction step. Most recent approaches use a U-Net [32]

architecture and a custom loss function

[22, 12]

. The remaining differences between methods can be found in the superpixel merging procedure. Several approaches are based on hierarchical clustering, but differ in how they accumulate boundary weights:

[22] use the accumulated mean CNN boundary predictions, [12]

employ quantile based accumulation and


re-predict the weights with a random forest

[5] after each agglomeration step. In contrast [1] and [3] solve a NP-hard graph partitioning problem, the (Lifted) Multicut. A notable exception from this three step approach is the flood filling network (FFN) [16] which can go directly from pixels to instances by predicting individual object masks one at a time.

Several groups have observed that these methods make segmentation errors which lead to morphologically impossible objects allowed by the initial CNN because of its limited field of view. This was first demonstrated by Rolnick et al. in [31], where a CNN is trained on downsampled segmentation masks to detect merge errors. However, they do not propose any method to fix the detected errors. In contrast, [26]

detect merge errors with a simple shape-based heuristic and then correct these with a FFN. Zung et al.

[42] were the first to combine CNN-based error detection and FFN-based correction. In their formulation both false merges and false splits can be corrected. Recently [9] have introduced an approach based on CNN error detection followed by a simple heuristic to correct false merges and lifted multicut graph partitioning to fix false splits.

Object morphology is not the only indicator of segmentation correctness. In EM data in particular, where the staining is indiscriminate and the cell ultrastructure is clearly visible, one can also test for biological plausibility of the contents of the segmented objects. Krasowski et al [18] demonstrated the validity of this strategy, exploiting the fact that in mammalian neural tissue axon and dendrite attribution should not be mixed for individual objects. They use the asymmetric multi-way cut (AMWC) [21], a generalization of the multicut for joint graph partition and node labeling, using axon / dendrite attribution as node labels.

Augmentation of the Multicut partitioning problem by lifted (long-range) edges in [15] introduced a principled way to incorporate long-range attractive/repulsive forces between the graph nodes directly into the segmentation procedure without resorting to post-processing heuristics. We exploited it for the neuron segmentation use case in [3], but did not express any domain knowledge – the weights of the lifted edges were learned directly from groundtruth segmentations and the edges themselves were added densely, but only over a graph distance of 2. The lifted edges in [3] made the segmentation algorithm more robust against single missing boundaries, but did not counter the problem of the limited field of view of the boundary CNN and did not prevent biologically implausible objects. Besides Lifted Multicut, the recently introduced Mutex Watershed can also exploit long-range information [39, 38].

While all the listed methods demonstrate increased segmentation accuracy, they do not offer a general recipe on how to exploit domain-specific knowledge in a segmentation algorithm. We propose a versatile framework that can incorporate such information from diverse sources by mapping it to sparse lifted edges in the lifted multicut objective.

3 Methods

Our method follows the three step segmentation approach described in section 2, starting from a boundary predictor and using graph partitioning to agglomerate super-pixels. First, we review the lifted multicut problem[15] in subsection 3.1. We follow by proposing a general approach to incorporate domain-specific knowledge into the lifted edges (subsection 3.2). Finally, we describe three specific applications with different sources of domain knowledge and show how our previous work on lifted multicut for neuron segmentation can be positioned in terms of the proposed framework.

3.1 Lifted Multicut Graph Partition

Instance segmentation can be formulated as a graph partition problem given a graph and edge weights . In our setting, the nodes correspond to fragments of the over-segmentation and edges link two nodes iff the two corresponding fragments share an image boundary. The weights

encode the attractive strength (positive values) or repulsive strength (negative values) of edges and are usually derived from (pseudo) probabilities

via negative log-likelihood:


The resulting partition problem is known as multicut or correlation clustering [6, 1, 8]. Its objective is given by


where are binary indicator variables linked to the edge state; 0 means that an edge connects the two adjacent nodes in the resulting partition, 1 means that it doesn’t. The constraints forbid dangling edges in the partition, i.e. edges that separate two nodes () for which a path of connecting edges () exists.

The lifted multicut [15] is an extension of the multicut problem, which introduces a new set of edges called lifted edges. These edges differ from regular graph edges by providing only an energy contribution, but not inducing connectivity. This is motivated by the observation that it is often helpful to derive non-local features for the connectivity of (super) pixels. The presence of an attractive non-local edge should not result in air bridges though, i.e. non-local edges that connect two pixels without a connection via local edges. In our setting, lifted edges connect nodes and that are not adjacent in . With the sets of original edges , lifted edges , binary indicator variables , and weights associated with all edges in the lifted multicut objective can be formulated as


The constraints (5) correspond to Equation 2 and enforce a consistent partition without dangling edges. Constraints (6) and (7) ensure that the state of lifted edges is consistent with the connectivity, i.e. that two nodes connected by a lifted edge are also connected via a path of regular edges and two nodes separated by a lifted edge are not connected by any such path.

Figure 2: (Left) Graph neighborhood of a single node (blue shaded segment) with local edges (blue lines) and dense lifted edges (orange dotted edges). (Right) Neighborhood with sparse lifted edges (yellow dotted edges), connecting to nodes with prior (red shaded segments).

3.2 Sparse Lifted Edges

Our main contribution is a general recipe how to express domain-specific knowledge via sparse lifted edges that are only added between graph nodes where attribution of this knowledge is possible. The right part of Figure 2 symbolizes this idea: nodes with attribution are symbolized by shaded segments and sparse lifted edges by yellow dashed lines.

The sparse lifted edges are constructed in several steps, see Figure 1. First, we compute the graph and edge weights from watersheds and boundary predictions. See (b) for attractive (green) and repulsive (red) regular graph edges. Then, we map the domain specific knowledge to nodes, see (c), and derive attractive and repulsive lifted edges, see green and red dashed lines in (d). The sign and strength of the lifted edge weights is derived by computing the likelihood of incident nodes being connected and then using Equation 1 to obtain signed weights. Finally, we solve the resulting lifted multicut objective to obtain an instance segmentation, see (e).

In our previous work [3], we have already applied the lifted multicut to neuron segmentation. There, we introduced lifted edges in a dense pattern connecting all nodes in a local graph neighborhood, see orange dashed edges in Figure 2. We computed lifted edge weights with a Random Forest based on features derived from region appearance and clustering dendrograms. Note that this approach can be understood as a special case of the framework proposed here, using generic, but weak knowledge about morphology and graph structure of segments.

Krasowski et al. [18] employ a similar strategy, in which they use the AMWC [21] graph partitioning and labeling formulation to incorporate domain specific knowledge. However, their approach has two drawbacks compared to our strategy. First, it is not possible to encode attractive information just with node labels. Second, it is harder to express information that does not fit the node labeling category, even if it is repulsive in nature. A good example for this is the morphology-based false merge detection. In this case, defining a labeling for only a subset of nodes is not possible, while we can add lifted edges only where false merge detections are present.

Axon Dendrite Attribution

In mammalian cortex, neurons have separate axonal and dendritic compartments that emerge from the soma in opposite directions. Given the existing size limitations in volumetric EM imaging (the largest block ever imaged measures to a cubic millimeter), we can thus infer the prior knowledge that axonal and dendritic attribution should not be mixed in a single segmented object. We exploit the fact that these attributes can be identified for parts of the data by vesicle clouds for axons and large shafts for dendrites. This allows to automatically detect axonal and dendritc compartments. We map these detections to graph nodes by overlap, see (c) in the left column of Figure 1. Blue shading indicates axon attribution, yellow shading dendrite attribution. Then we add lifted edges between nodes mapped to different classes, see dashed lines in (e), and infer the edge weights from attribution statistics. See subsection 4.1 for details on this problem set-up and results.

False Merge Detector

For neurons in the insect brain the axon/dendrite separation is not pronounced and the approach described in the previous section not applicable. Instead, we can leverage morphological information for individual segmented objects to identify and resolve segmentation errors. Here, we use a random forest based on features extracted from paths within the segmented objects to detect false merges. Note that more sophisticated approaches for this task have been proposed, see

section 2. Our resolving method based on sparse lifted edges would work with these error detectors as well and most likely yield better results due to more informative priors. We restrict ourselves to a simple false merge detector since the focus of this work is not the detector itself, but the framework of exploiting its predictions in an automated manner.

Our classifier predicts whether the paths inside of segmented objects contain a false merge. We select all objects containing at least one path with the false merge probability larger than a preset threshold for resolving. For these objects, we add lifted edges between pairs of nodes corresponding to the terminals of the paths used for error prediction, deriving the weights from the path’s false merge probabilities.

Figure 1 shows an example false merger as a blue shaded object in (c) and the lifted edges derived from the corresponding paths in (d). We describe the experiment set-up in more detail and show quantitative results in subsection 4.2.

Sparse semantic labels

In this example, we tackle a segmentation problem in sponge choanocyte chambers. These structures are built from several surrounding cells, the choanocytes, that interact with the central cell via flagella which are surrounded by a collar of microvilli. Our goal is to segment cell bodies, flagella and microvilli. This task is challenging due to the large difference in sizes of these structures. Especially the segmentation of the small flagella and microvilli is difficult. Without the use of domain specific knowledge they are split into many pieces because of insufficient feature statistics due to small image boundaries.

In order to alleviate these false split errors, we automatically predict flagella and microvilli attribution. We then compute an instance segmentation based on these predictions by applying a threshold and connected components. We map the component labels to nodes of the graph, right column in Figure 1, panel(c). Then, we introduce attractive lifted edges between the nodes that were mapped to the same component label and repulsive lifted edges between nodes mapped to different component labels, see (d). Rather than deriving the lifted edge weights from data, we use preset values for attractiveness / repulsiveness. See subsection 4.3 for more details and results.

3.3 Hierarchical Lifted Multicut Solver

Finding the optimal solution of the lifted multicut objective is NP-hard. Approximate solvers based on greedy algorithms [17] and fusion moves [3] have been introduced. However, even these approximations do not scale to the large problem we need to solve in the sponge segmentation example. In order to tackle this and even larger problems, we adapt the hierarchical multicut solver of [30] for lifted multicuts.

This solver extracts sub-problems from a regular tiling of the volume, solves these sub-problems in parallel and uses the solutions to contract nodes in the graph, thus reducing its size. This approach can be repeated for an increasing size of the blocks that are used to tile the volume, until the reduced problem becomes feasible with an other (approximate) solver.

We extend this approach to the lifted multicut by also extracting lifted edges during the sub-problem extraction. We only extract lifted edges that connect nodes in the sub-graph defined by the block at hand. This strategy, where we ignore lifted edges crossing block boundaries, is in line with the idea that lifted edges contribute to the energy, but not to the connectivity. Note that lifted edges that are not part of any sub-problem at a given level will still be considered at a later stage. See algorithm 1 for pseudo-code.

Note that this approach is conceptually similar to the fusion move based approximation of [4], which extracts and solves sub-problems based on a random graph partition and accepts changes from sub-solutions if they increase the overall energy, repeating this process until convergence. Compared to this approach, we extract sub-problems from a deterministic partition of the graph. This allows us to solve only a preset number of sub-problems leading to faster convergence.

Note that our approximate solver is only applicable if the graph at hand has a spatial embedding, which allows to extract sub-problems from a tiling of space. In our case, this spatial embedding is given by the watershed fragments that correspond to nodes.

4 Results

We study the performance of the proposed method on three different problems: instance segmentation in murine cortex with priors from axon/dendrite segmentation, instance segmentation in drosophila brain with priors from morphology-based error detection and instance segmentation in a sponge choanocyte chamber with priors from semantic segmentation. See Table 3 for an overview of the different problem set-ups. We evaluate segmentation quality using the variation of information (VI) [25], which can be separated into split and merge scores, and the adapted rand score [2].


4.1 Axon Dendrite Attribution

We present results on a volume of murine somatosensory cortex that was acquired by FIBSEM at 5 5 6 nanometer resolution. The same volume has already been used in [18] for a similar experiment. To ensure a fair comparison between the two methods for incorporating axon/dendrite priors, we obtained derived data from the authors and use it to set-up the segmentation problem.

This derived data includes probability maps for cell membrane, mitochondria, axon and dendrite attribution as well as a watershed over-segmentation derived from the cell membrane probabilities and ground-truth instance segmentation. From this data, we set up the graph partition problem as follows: we build the region adjacency graph from the watersheds and compute weights for the regular edges with a random forest based on edge and region appearance features. See [3] for a detailed description of the feature set. Next, we introduce dense lifted edges up to a graph distance of three. We use a random forest based on features derived from region appearance and clustering to predict their weights, see [3] for details. In addition to the region appearance features only based on raw data, we also take into account the mitochondria attribution here. Next, we map the axon/dendrite attribution to the nodes of and introduce sparse lifted edges between nodes mapped to different classes. We infer weights for these edges with a random forest based on features from the statistics of the axon and dendrite node mapping. We use the fusion move solver of [4] for optimizing the lifted multicut objective.

We divide the volume into a 1 3.5 3.5 micron block that is used to train the random forests for edge weights and a 2.5 3.5 3.5 micron block used for evaluation. The random forest predicting pixel-wise probabilities was trained by the authors of [18] on a separate volume, using ilastik [34].

We compare the multicut and AMWC solutions reported in [18] with different variants of our methods, see Table 1. As a baseline, we compute the lifted multicut only with dense lifted edges and without features from mitochondria predictions (LMC-D). We compute the full model with dense and sparse lifteds (LMC-S) with and without additional features for dense lifted edges from mitochondria predictions. In addition, we compare to an iterative approach (LMC-SI) similar to the error correction approach in subsection 4.2, where we perform LMC-D segmentation first and introduce sparse lifted edges only for objects that contain a false merge (identified by presence of both axonic and dendritic nodes in the same object).

The LMC-D segmentation quality is on par with the AMWC, although it does not use any input from the priors, showing the importance of dense lifted edges. Our full model with sparse lifted edges shows significantly better quality compared to LMC-D. Mitochondria-based features provide a small additional boost. The segmentation quality of the iterative approach LMC-SI is inferior to solving the full model LMC-S. This shows the importance of joint optimization of the full model with dense and sparse lifted edges.

Method VI-Split VI-Merge Rand Error
MC [18] 0.3471 0.6347 0.0787
AMWC [18] 0.4578 0.4935 0.0754
LMC-D 0.4144 0.4445 0.0891
LMC-S 0.4133 0.3788 0.0362
LMC-S (No Mitos) 0.4038 0.3966 0.0363
LMC-SI 0.5054 0.3998 0.0586
Table 1: Variants of our approach compared to the method of [18]. The Rand Error measures the over-all segmentation quality, while VI-Split measures the degree of over-segmentation and VI-Merge the degree of under-segmentation. For all measures, a lower score corresponds to a better segmentation.

4.2 False Merge Detector

We test the false merge correction on parts of the Drosophila medulla. This data was made public by the authors of [35], who also provide membrane probability maps, a watershed over-segmentation and a ground-truth segmentation for the whole volume.

In order to demonstrate segmentation improvement based on morphological knowledge, we train a random forest that can detect false merges in segmented objects. This classifier is based on morphological features derived from paths inside of segmented objects. These paths are extracted from skeletons computed with the method of [23] using the implementation from [36]. See section 7 for a detailed description of the feature set. The random forest is trained based on correct and falsely merged objects from an initial multicut segmentation of a separate part of the dataset.

Given this false merge detector, we set up the lifted multicut problem as follows: we compute the region adjacency graph based on the watersheds and compute weights for the regular edges with a random forest based on the edge and region appearance features described in [3]. We add dense lifted edges up to a graph neighborhood of three and compute weights based on region appearance and clustering, again following [3]. We solve this lifted multicut (LMC-D) to obtain an initial segmentation and extract paths and features from it to apply the error detector. We introduce sparse lifted edges for all objects that contain at least one path with false merge probability larger than 0.5. Note that we introduce lifted edges corresponding to all paths in such an object and derive attractive and repulsive weights in order to improve the merge localisation. We then resolve these objects with a lifted multicut problem set up from regular edges as well as dense and sparse lifted edges (LMC-SI). We use the fusion move solver of [4] to optimize the resulting lifted multicut objective.

We train the random forest for regular and dense lifted edges on a 2 2 2 micron cube and train the random forest for false merge detection on five 5 5 5 micron cubes cut out of the full volume. For inference, we cut out a 10 10

10 micron cube and split it into 8 sub-volumes. We run inference on these sub-volumes with LMC-D and then re-solve the objects predicted as false merges with with LMC-SI. To estimate the upper limit of the performance of the proposed method, we run another experiment using an oracle predictor (always returning correct weights for the sparse lifted edges) instead of the error detection random forest. We refer to this approach as LMC-SO. The results for all blocks are shown in

Figure 3. The raw numbers behind this figure can be found in Table 4

, along with the F-scores of the false merge classifier. We observe a strong improvement of the VI merge scores for all blocks. Predictably, the number of false split errors goes up, but overall the LMC-SI segmentation is better as shown by the improvement of the Rand Score in

Table 4. Figure 4 shows an example LMC-SI result: the 5 objects visible in the figure were all merged into one by LMC-D and resolved by LMC-SI.

Figure 3: VI values for LMC-D, LMC-SI and LMC-SO segmentation on 8 test blocks.
Figure 4: Object corrected by LMC-SI, see Figure 6 for more details.

4.3 Sponge Choanocyte Segmentation

The two previous experiments mostly profited from repulsive information derived from ultrastructure or morphology. In order to show how attractive information can be exploited, we turn to an instance segmentation problem in a sponge choanocyte chamber. The EM volume was imaged with FIBSEM at a resolution of 15 15 15 nanometer. We aim to segment structures of three different types: cell bodies, flagella and microvilli. Flagella and microvilli have a small diameter, which make them difficult to segment with a boundary based approach. On the other hand, cell bodies have a much larger diameter and touch each other, which makes a boundary based approach appropriate.

In order to set-up the segmentation problem, we first compute probability maps for boundaries, microvilli and flagella attribution using the autocontext workflow of ilastik [34]. We set-up the lifted multicut problem by first computing watersheds based on boundary maps, extracting the region adjacency graph and computing regular edge weights from the boundary maps accumulated over the edge pixels. We do not introduce dense lifted edges. For sparse lifted edges, we compute an additional instance segmentation of flagella and microvilli by thresholding the corresponding probability maps and running connected components. Then, we map the components of this segmentation to graph nodes and connect nodes mapped to the same component via attractive lifted edges and nodes mapped to different components via repulsive lifted edges. We use the hierarchical lifted multicut solver introduced in subsection 3.3 to solve the resulting objective, using the approximate solver of [17] to solve sub-problems. Note that the full model contained too many variables to be optimized by any other solver in a reasonable amount of time.

We run our segmentation approach on the whole volume, which covers a volume of 70 75 50 microns, corresponding to 4600 5000 3300 voxels. For evaluation, we use three cutouts of size 15 15 1.5 microns with ground-truth for instance and semantic segmentation. We split the evaluation into separate scores for objects belonging to the three different structures, extracting them based on the semantic segmentation ground-truth. See Table 2 for the evaluation results, comparing the sparse lifted multicut (LMC) to the multicut baseline (MC). As expected the quality of the segmentation of cell bodies is not affected, because we don’t introduce lifted edges for those. The split rate in flagella and microvilli decreases significantly leading to a better overall segmentation for these structures.

Method VI-Split VI-merge Rand Error
MC 0.6058 0.0116 0.0783
LMC 0.6004 0.0116 0.0782
MC 0.4728 0.0812 0.1205
LMC 0.2855 0.0812 0.0429
MC 3.1760 1.1101 0.7409
LMC 2.2745 1.1807 0.6973
Table 2: Quality of the sponge chonanocyte segmentation for the three different types of structures.

5 Discussion

We propose a general purpose strategy to leverage domain-specific knowledge for instance segmentation problems arising from EM image analysis. This strategy makes use of a graph partitioning problem known as lifted multicut by expressing the domain knowledge in the long-range lifted edges. We apply the proposed strategy to a diverse set of EM instance segmentation problems and consistently show an improvement in segmentation accuracy. For an application with ultrastructure based priors, we also observe that the lifted multicut based formulation yields higher quality results than the AMWC formulation of [18]. We believe that this is due to joint exploitation of dense short-range and sparse long-range information. A complete joint solution, with both lifted edges and semantic labels, has recently been introduced in [24]. We look forward to exploring the potential of this objective for the neuron segmentation problem.

Similar to the findings of [21], we demonstrated that prevention of merge errors is more efficient than their correction: the joint solution of LMC-S is more accurate than iterative LMC-SI. However, not all prior information can be incorporated directly into the original segmentation problem. For these priors we demonstrate how to construct an additional resolving step which can also significantly reduce the number of false merge errors. In the future we plan to further improve our segmentations by other sources of information: matches of the segmented objects to known cell types, manual skeletons or correlative light microscopy imaging.


  • [1] B. Andres et al. Globally optimal closed-surface segmentation for connectomics. In ECCV, 2012.
  • [2] I. Arganda-Carreras, S. C. Turaga, D. R. Berger, D. Cireşan, A. Giusti, L. M. Gambardella, J. Schmidhuber, D. Laptev, S. Dwivedi, J. M. Buhmann, et al. Crowdsourcing the creation of image segmentation algorithms for connectomics. Frontiers in neuroanatomy, 9:142, 2015.
  • [3] T. Beier et al. Multicut brings automated neurite segmentation closer to human performance. Nature Methods, 14(2), 2017.
  • [4] T. Beier and others. An efficient fusion move algorithm for the minimum cost lifted multicut problem. In ECCV, 2016.
  • [5] L. Breiman. Random forests. Machine learning, 45(1):5–32, 2001.
  • [6] S. Chopra et al. The partition problem. Mathematical Programming, 59(1-3), 1993.
  • [7] D. Ciresan, A. Giusti, L. M. Gambardella, and J. Schmidhuber. Deep neural networks segment neuronal membranes in electron microscopy images. In Advances in neural information processing systems, pages 2843–2851, 2012.
  • [8] E. D. Demaine, D. Emanuel, A. Fiat, and N. Immorlica. Correlation clustering in general weighted graphs. Theoretical Computer Science, 361(2-3):172–187, 2006.
  • [9] K. Dmitriev12, T. Parag, B. Matejek, A. E. Kaufman, and H. Pfister. Efficient correction for em connectomics with skeletal representation. In British Machine Vision Conferemce (BMVC), 2018.
  • [10] A. Eberle, S. Mikula, R. Schalek, J. Lichtman, M. K. Tate, and D. Zeidler. High-resolution, high-throughput imaging with a multibeam scanning electron microscope. Journal of microscopy, 259(2):114–120, 2015.
  • [11] K. Eichler et al. The complete connectome of a learning and memory centre in an insect brain. Nature, 548(7666), 2017.
  • [12] J. Funke, F. D. Tschopp, W. Grisaitis, A. Sheridan, C. Singh, S. Saalfeld, and S. C. Turaga. Large scale image segmentation with structured loss based deep learning for connectome reconstruction. IEEE transactions on pattern analysis and machine intelligence, 2018.
  • [13] K. J. Hayworth, C. S. Xu, Z. Lu, G. W. Knott, R. D. Fetter, J. C. Tapia, J. W. Lichtman, and H. F. Hess. Ultrastructurally smooth thick partitioning and volume stitching for large-scale connectomics. Nature methods, 12(4):319, 2015.
  • [14] L. Heinrich, J. Funke, C. Pape, J. Nunez-Iglesias, and S. Saalfeld. Synaptic cleft segmentation in non-isotropic volume electron microscopy of the complete drosophila brain. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 317–325. Springer, 2018.
  • [15] A. Horňáková et al. Analysis and optimization of graph decompositions by lifted multicuts. In ICML, 2017.
  • [16] M. Januszewski, J. Kornfeld, P. H. Li, A. Pope, T. Blakely, L. Lindsey, J. Maitin-Shepard, M. Tyka, W. Denk, and V. Jain. High-precision automated reconstruction of neurons with flood-filling networks. Nature methods, 15(8):605, 2018.
  • [17] M. Keuper et al. Efficient decomposition of image and mesh graphs by lifted multicuts. In ICCV, 2015.
  • [18] N. Krasowski et al. Neuron segmentation with high-level biological priors. IEEE transactions on medical imaging, 2017.
  • [19] A. Kreshuk, U. Koethe, E. Pax, D. D. Bock, and F. A. Hamprecht. Automated detection of synapses in serial section transmission electron microscopy image stacks. PloS one, 9(2):e87351, 2014.
  • [20] A. Kreshuk, C. N. Straehle, C. Sommer, U. Koethe, M. Cantoni, G. Knott, and F. A. Hamprecht. Automated detection and segmentation of synaptic contacts in nearly isotropic serial electron microscopy images. PloS one, 6(10):e24899, 2011.
  • [21] T. Kroeger et al. Asymmetric cuts: Joint image labeling and partitioning. In GCVPR, 2014.
  • [22] K. Lee et al. Superhuman accuracy on the snemi3d connectomics challenge. arXiv, 2017.
  • [23] T.-C. Lee et al. Building skeleton models via 3-d medial surface axis thinning algorithms. CVGIP: Graphical Models and Image Processing, 1994.
  • [24] E. Levinkov et al. Joint graph decomposition & node labeling: Problem, algorithms, applications. In CVPR, 2017.
  • [25] M. Meilă. Comparing clusterings by the variation of information. In Learning theory and kernel machines. 2003.
  • [26] Y. Meirovitch et al. A multi-pass approach to large-scale connectomics. arXiv, 2016.
  • [27] J. Nixon-Abell, C. J. Obara, A. V. Weigel, D. Li, W. R. Legant, C. S. Xu, H. A. Pasolli, K. Harvey, H. F. Hess, E. Betzig, et al. Increased spatiotemporal resolution reveals highly dynamic dense tubular matrices in the peripheral er. Science, 354(6311):aaf3928, 2016.
  • [28] J. Nunez-Iglesias, R. Kennedy, T. Parag, J. Shi, and D. B. Chklovskii. Machine learning of hierarchical clustering to segment 2d and 3d images. PloS one, 8(8):e71715, 2013.
  • [29] S. Otsuka, A. M. Steyer, M. Schorb, J.-K. Heriche, M. J. Hossain, S. Sethi, M. Kueblbeck, Y. Schwab, M. Beck, and J. Ellenberg. Postmitotic nuclear pore assembly proceeds by radial dilation of small membrane openings. Nature structural & molecular biology, 25(1):21, 2018.
  • [30] C. Pape et al. Solving large multicut problems for connectomics via domain decomposition. In ICCV, 2017.
  • [31] D. Rolnick et al. Morphological error detection in 3d segmentations. arXiv, 2017.
  • [32] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  • [33] M. R. Russell, T. R. Lerner, J. J. Burden, D. O. Nkwe, A. Pelchen-Matthews, M.-C. Domart, J. Durgan, A. Weston, M. L. Jones, C. J. Peddie, et al. 3d correlative light and electron microscopy of cultured cells using serial blockface scanning electron microscopy. J Cell Sci, 130(1):278–291, 2017.
  • [34] C. Sommer, C. Straehle, U. Koethe, and F. A. Hamprecht. Ilastik: Interactive learning and segmentation toolkit. In 2011 IEEE international symposium on biomedical imaging: From nano to macro, pages 230–233. IEEE, 2011.
  • [35] S. Takemura et al. Synaptic circuits and their variations within different columns in the visual system of drosophila. PNAS, 112(44), 2015.
  • [36] S. Van der Walt, J. L. Schönberger, J. Nunez-Iglesias, F. Boulogne, J. D. Warner, N. Yager, E. Gouillart, and T. Yu. scikit-image: image processing in python. PeerJ, 2:e453, 2014.
  • [37] J. G. White, E. Southgate, J. N. Thomson, and S. Brenner. The structure of the nervous system of the nematode caenorhabditis elegans. Philos Trans R Soc Lond B Biol Sci, 314(1165):1–340, 1986.
  • [38] S. Wolf, A. Bailoni, C. Pape, N. Rahaman, A. Kreshuk, U. Köthe, and F. A. Hamprecht. The mutex watershed and its objective: Efficient, parameter-free image partitioning. arXiv preprint arXiv:1904.12654, 2019.
  • [39] S. Wolf, C. Pape, A. Bailoni, N. Rahaman, A. Kreshuk, U. Kothe, and F. Hamprecht. The mutex watershed: efficient, parameter-free image partitioning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 546–562, 2018.
  • [40] C. Xiao, W. Li, X. Chen, H. Han, and Q. Xie. An effective fully deep convolutional neural network for mitochondria segmentation based on atum-sem. In Medical Imaging 2018: Image Processing, volume 10574, page 105740C. International Society for Optics and Photonics, 2018.
  • [41] Z. Zheng, J. S. Lauritzen, E. Perlman, C. G. Robinson, M. Nichols, D. Milkie, O. Torrens, J. Price, C. B. Fisher, N. Sharifi, et al. A complete electron microscopy volume of the brain of adult drosophila melanogaster. Cell, 174(3):730–743, 2018.
  • [42] J. Zung, I. Tartavull, K. Lee, and H. S. Seung. An error detection and correction framework for connectomics. In Advances in Neural Information Processing Systems, pages 6818–6829, 2017.

6 Appendix 1: Hierarchical Lifted Multciut Solver

Data: graph , edge weights , lifted edges and weights and , nLevels, blockShape
Result: node partition
= ;
for n in nLevels do
      1 blocks = getBlocks(blockShape);
       subPartitions = [];
       /* this for-loop can be parallelized */
       for block in blocks do
            2 = getSubproblem(, , block);
            3 = getLiftedEdges(, , );
            4 = solveLiftedMulticut();
       end for
      5 = reduceProblem(, subPartitions);
       blockShape *= 2;
end for
= solveLiftedMulticut();
= projectToInitialGraph();
Algorithm 1 Hierarchical lifted multicut algorithm based on the approximate multicut solver of [30]. (1): getBlocks tiles the volume with blocks of blockShape. (2): getSubproblem extracts the sub-graph and weights of edges in this graph from the given block coordinates. (3): getLiftedEdges extracts the lifted edges that connect nodes which are both part of the sub-graph as well as the corresponding weights. (4): solveLiftedMulticut solves the lifted multicut problem using one of the two approximate solvers [3, 17]. (5): reduceProblem: reduces the graph by contracting nodes according to the sub-partition results. Also updates edge weights as well as lifted edges and their weights accordingly.

7 Appendix 2: Skeleton Path Features

To extract features from skeleton paths, we take into account three different input maps: the raw data, the membrane probabilities and the distance transform to the boundary of the object from which the skelleton was extracted. We apply the following image filter on these inputs: Gaussian Smoothing, Laplacian of Gaussian, Hessian of Gaussian Eigenvalues. Each filter is applied with three different sigma values: 1.6, 4.2 and 8.4. To obtain features, we accumulate the values of these inputs over the skeleton path’s pixel and apply the following statistics: Mean, Variance, Skewness, Kurtosis, Quantiles.

8 Appendix 3: False Merge Correction Results

Block 1 2 3 4 5 6 7 8
VI-S 0.421 0.531 0.561 0.425 0.480 0.458 0.491 0.468
VI-M 0.766 0.996 0.797 0.745 0.753 0.818 0.853 0.812
RAND 0.137 0.277 0.223 0.152 0.178 0.161 0.194 0.165
VI-S 0.555 0.722 0.723 0.602 0.633 0.641 0.633 0.628
VI-M 0.598 0.752 0.651 0.572 0.581 0.646 0.694 0.630
RAND 0.117 0.241 0.227 0.143 0.152 0.151 0.174 0.146
VI-S 0.457 0.624 0.605 0.491 0.530 0.519 0.536 0.525
VI-M 0.494 0.642 0.548 0.480 0.489 0.518 0.532 0.523
RAND 0.078 0.200 0.179 0.098 0.114 0.100 0.099 0.101
Table 4: Results of initial segmentation with LMC-D and false merge correction with LMC-SI and LMC-SO

We include additional material on the false merge correction results: the table underlying the results graphic, Table 4, evaluation curves for the random forest we have trained for false merge path classification Figure 5 and a more detailed figure of the example object shown in the results. Figure 6

1 Block1
2 Block2
3 Block3
4 Block4
5 Block5
6 Block6
7 Block7
8 Block8
Figure 5:

F-Score, accuracy, precision and recall curves for the 8 test blocks.

1 Initial, falsely merged object with skeleton paths.
2 Resolved object.
3 Ground-truth object.
Figure 6: Object from test-block 2 that was correctly resolved by LMC-SI.