1 Introduction
Largescale electron microscopy (EM) imaging is becoming an increasingly important tool in different fields of biology. The technique was pioneered by efforts to map the neural circuitry of small animals at synaptic resolution to obtain their socalled connectome. In the 1980’s White et al. [37] have mapped the complete connectome of C. elegans in a manual tracing effort which spanned over a decade. Since then, the throughput of EM imaging has increased by several orders of magnitude thanks to innovations like multibeam serial section EM [10], hotknife stitching [13] or gas cluster milling [13]. This allows to image much larger volumes up to the complete brain of the fruitfly larva [11] and even the adult fruitfly [41]. Recently, studies based on largescale EM have become more common in other fields of biology as well [27, 29, 33].
Given the enormous amount of data generated, automated analysis of the acquired images is crucial; one of the key steps being instance segmentation of cells, neurons or cellular organelles. In recent years, the accuracy of automated segmentation algorithms has increased significantly thanks to the rise of deep learningbased tools in computer vision and the development of convolutional neural networks (CNNs) for semantic and instance segmentation
[3, 12, 22, 16]. Still, it is not yet good enough to completely forego human proofreading. Out of all EM image analysis problems, neuron segmentation turned out to be particularly difficult[16] due to the small diameter and long reach of neurons and astrocytes, but other EM segmentation problems have not yet been fully automated either. One of the major downsides of CNNbased segmentation approaches lies in their limited field of view which makes them overly reliant on local boundary evidence. Heavy metal staining used in the EM sample preparation stains all cells indiscriminately and forces segmentation algorithms to rely on cell membrane detection to separate membranebound objects. Staining artifacts or alignment issues can severely weaken this evidence and often cause false merge errors where separate objects get merged into one. On the other hand, membranes of cellular organelles or objects of small diameter often cause false split errors where a single structure gets split into several segmented objects.Human experts avoid many of these errors by exploiting additional prior knowledge on the expected object shape or constraints from higherlevel biology. Following this observation, several algorithms have recently been introduced to enable detection of morphological errors in segmented objects [31, 42, 9]. By looking at complete objects rather than a handful of pixels, these algorithms can significantly improve the accuracy of the initial segmentation. In addition to purely morphological criteria, Krasowski et al. in [18] suggested an algorithm to exploit biological priors such as an incompatible mix of ultrastructure elements.
Building on such prior work, this contribution introduces a general approach to leverage domainspecific knowledge for the improvement of segmentation accuracy. It allows to incorporate a large variety of rules, explicit or learned from data, which can be expressed as the likelihood of certain areas in the image to belong to the same object in the segmentation. The areas can be sparse and/or spatially distant. In more detail, we formulate the segmentation problem with such rules as a graph partitioning problem with longrange attractive or repulsive edges.
Horvnakova et al. in [15] showed that this problem, which they term Lifted Multicut as it corresponds to the Multicut partitioning problem with additional lifted (longrange) edges, can be solved in reasonable time for small problem sizes, while Beier et al. in [4] introduced an efficient approximate solver. Lifted Multicut with dense, shortrange lifted edges has already been applied to the neuron segmentation problem [3]. There, the weights of the lifted edges were learned directly from the groundtruth segmentations. While the introduction of dense, shortrange lifted edges already improves segmentation in comparison to regular Multicut partitioning, it does not allow to express explicit or learned biological knowledge on the complex object morphology or on the relations between different semantic entities. Exploiting such knowledge is the target of this contribution.
For the problem of image segmentation, the graph in the Multicut partitioning problem corresponds to the region adjacency graph of the image pixels or superpixels. The nodes of the graph can be mapped directly to spatial locations in the image. Domain knowledge expressed as rules that certain locations must or must not belong to the same object can thus be distilled into lifted edges between these locations. The weights of such lifted edges are derived from the strictness of the rules, which can colloquially range from ”usually do / do not belong to the same object” to ”always / never ever belong to the same object”.
We demonstrate the validity of this approach for three applications, incorporating different kinds of domain knowledge into the lifted multicut framework:

using indicators of axon/dendrite attribution to avoid merges between axonal and dendritic neural processes,

using learned object morphology to detect and correct falsely merged neural processes and

using semantic segmentation of biological structures to decrease the number of false splits in instance segmentation of sponge choanocytes.
Figure 1 gives an overview of the three applications. For all three, additional domain knowledge allows us to significantly improve segmentation accuracy. Aiming to apply the method to data of biologically relevant size, we addtionally introduce a new scalable solver for the lifted multicut problem based on our prior work from [30].
2 Related Work
Instance segmentation problems that arise in EM image analysis can be roughly separated into two categories: segmenting structures with background that touch rarely, such as synapses [20, 19, 14] or mitochondria[40] and segmenting large structures with little or no background, like neural processes or cells. For the first set of problems semantic segmentation or detection based approaches are most common, while the second is usually tackled with boundarybased instance segmentation. Here, we focus on the second usecase, but show that we can also bridge both categories in subsection 4.3.
Neuron segmentation of connectomics has been the main driver of the recent advances in boundarybased EM segmentation. Most methods ([1, 3, 28, 12, 22]) follow a three step procedure: in the first step they segment boundaries, then compute a superpixel oversegmentation and finally combine the superpixels into objects. The success of a CNN [7] in the ISBI 2012 neuron segmentation challenge [2] has prompted the adoption of this technique for the boundary prediction step. Most recent approaches use a UNet [32]
architecture and a custom loss function
[22, 12]. The remaining differences between methods can be found in the superpixel merging procedure. Several approaches are based on hierarchical clustering, but differ in how they accumulate boundary weights:
[22] use the accumulated mean CNN boundary predictions, [12]employ quantile based accumulation and
[28]repredict the weights with a random forest
[5] after each agglomeration step. In contrast [1] and [3] solve a NPhard graph partitioning problem, the (Lifted) Multicut. A notable exception from this three step approach is the flood filling network (FFN) [16] which can go directly from pixels to instances by predicting individual object masks one at a time.Several groups have observed that these methods make segmentation errors which lead to morphologically impossible objects allowed by the initial CNN because of its limited field of view. This was first demonstrated by Rolnick et al. in [31], where a CNN is trained on downsampled segmentation masks to detect merge errors. However, they do not propose any method to fix the detected errors. In contrast, [26]
detect merge errors with a simple shapebased heuristic and then correct these with a FFN. Zung et al.
[42] were the first to combine CNNbased error detection and FFNbased correction. In their formulation both false merges and false splits can be corrected. Recently [9] have introduced an approach based on CNN error detection followed by a simple heuristic to correct false merges and lifted multicut graph partitioning to fix false splits.Object morphology is not the only indicator of segmentation correctness. In EM data in particular, where the staining is indiscriminate and the cell ultrastructure is clearly visible, one can also test for biological plausibility of the contents of the segmented objects. Krasowski et al [18] demonstrated the validity of this strategy, exploiting the fact that in mammalian neural tissue axon and dendrite attribution should not be mixed for individual objects. They use the asymmetric multiway cut (AMWC) [21], a generalization of the multicut for joint graph partition and node labeling, using axon / dendrite attribution as node labels.
Augmentation of the Multicut partitioning problem by lifted (longrange) edges in [15] introduced a principled way to incorporate longrange attractive/repulsive forces between the graph nodes directly into the segmentation procedure without resorting to postprocessing heuristics. We exploited it for the neuron segmentation use case in [3], but did not express any domain knowledge – the weights of the lifted edges were learned directly from groundtruth segmentations and the edges themselves were added densely, but only over a graph distance of 2. The lifted edges in [3] made the segmentation algorithm more robust against single missing boundaries, but did not counter the problem of the limited field of view of the boundary CNN and did not prevent biologically implausible objects. Besides Lifted Multicut, the recently introduced Mutex Watershed can also exploit longrange information [39, 38].
While all the listed methods demonstrate increased segmentation accuracy, they do not offer a general recipe on how to exploit domainspecific knowledge in a segmentation algorithm. We propose a versatile framework that can incorporate such information from diverse sources by mapping it to sparse lifted edges in the lifted multicut objective.
3 Methods
Our method follows the three step segmentation approach described in section 2, starting from a boundary predictor and using graph partitioning to agglomerate superpixels. First, we review the lifted multicut problem[15] in subsection 3.1. We follow by proposing a general approach to incorporate domainspecific knowledge into the lifted edges (subsection 3.2). Finally, we describe three specific applications with different sources of domain knowledge and show how our previous work on lifted multicut for neuron segmentation can be positioned in terms of the proposed framework.
3.1 Lifted Multicut Graph Partition
Instance segmentation can be formulated as a graph partition problem given a graph and edge weights . In our setting, the nodes correspond to fragments of the oversegmentation and edges link two nodes iff the two corresponding fragments share an image boundary. The weights
encode the attractive strength (positive values) or repulsive strength (negative values) of edges and are usually derived from (pseudo) probabilities
via negative loglikelihood:(1) 
The resulting partition problem is known as multicut or correlation clustering [6, 1, 8]. Its objective is given by
(2)  
(3) 
where are binary indicator variables linked to the edge state; 0 means that an edge connects the two adjacent nodes in the resulting partition, 1 means that it doesn’t. The constraints forbid dangling edges in the partition, i.e. edges that separate two nodes () for which a path of connecting edges () exists.
The lifted multicut [15] is an extension of the multicut problem, which introduces a new set of edges called lifted edges. These edges differ from regular graph edges by providing only an energy contribution, but not inducing connectivity. This is motivated by the observation that it is often helpful to derive nonlocal features for the connectivity of (super) pixels. The presence of an attractive nonlocal edge should not result in air bridges though, i.e. nonlocal edges that connect two pixels without a connection via local edges. In our setting, lifted edges connect nodes and that are not adjacent in . With the sets of original edges , lifted edges , binary indicator variables , and weights associated with all edges in the lifted multicut objective can be formulated as
(4)  
(5)  
(6)  
(7) 
The constraints (5) correspond to Equation 2 and enforce a consistent partition without dangling edges. Constraints (6) and (7) ensure that the state of lifted edges is consistent with the connectivity, i.e. that two nodes connected by a lifted edge are also connected via a path of regular edges and two nodes separated by a lifted edge are not connected by any such path.
3.2 Sparse Lifted Edges
Our main contribution is a general recipe how to express domainspecific knowledge via sparse lifted edges that are only added between graph nodes where attribution of this knowledge is possible. The right part of Figure 2 symbolizes this idea: nodes with attribution are symbolized by shaded segments and sparse lifted edges by yellow dashed lines.
The sparse lifted edges are constructed in several steps, see Figure 1. First, we compute the graph and edge weights from watersheds and boundary predictions. See (b) for attractive (green) and repulsive (red) regular graph edges. Then, we map the domain specific knowledge to nodes, see (c), and derive attractive and repulsive lifted edges, see green and red dashed lines in (d). The sign and strength of the lifted edge weights is derived by computing the likelihood of incident nodes being connected and then using Equation 1 to obtain signed weights. Finally, we solve the resulting lifted multicut objective to obtain an instance segmentation, see (e).
In our previous work [3], we have already applied the lifted multicut to neuron segmentation. There, we introduced lifted edges in a dense pattern connecting all nodes in a local graph neighborhood, see orange dashed edges in Figure 2. We computed lifted edge weights with a Random Forest based on features derived from region appearance and clustering dendrograms. Note that this approach can be understood as a special case of the framework proposed here, using generic, but weak knowledge about morphology and graph structure of segments.
Krasowski et al. [18] employ a similar strategy, in which they use the AMWC [21] graph partitioning and labeling formulation to incorporate domain specific knowledge. However, their approach has two drawbacks compared to our strategy. First, it is not possible to encode attractive information just with node labels. Second, it is harder to express information that does not fit the node labeling category, even if it is repulsive in nature. A good example for this is the morphologybased false merge detection. In this case, defining a labeling for only a subset of nodes is not possible, while we can add lifted edges only where false merge detections are present.
Axon Dendrite Attribution
In mammalian cortex, neurons have separate axonal and dendritic compartments that emerge from the soma in opposite directions. Given the existing size limitations in volumetric EM imaging (the largest block ever imaged measures to a cubic millimeter), we can thus infer the prior knowledge that axonal and dendritic attribution should not be mixed in a single segmented object. We exploit the fact that these attributes can be identified for parts of the data by vesicle clouds for axons and large shafts for dendrites. This allows to automatically detect axonal and dendritc compartments. We map these detections to graph nodes by overlap, see (c) in the left column of Figure 1. Blue shading indicates axon attribution, yellow shading dendrite attribution. Then we add lifted edges between nodes mapped to different classes, see dashed lines in (e), and infer the edge weights from attribution statistics. See subsection 4.1 for details on this problem setup and results.
False Merge Detector
For neurons in the insect brain the axon/dendrite separation is not pronounced and the approach described in the previous section not applicable. Instead, we can leverage morphological information for individual segmented objects to identify and resolve segmentation errors. Here, we use a random forest based on features extracted from paths within the segmented objects to detect false merges. Note that more sophisticated approaches for this task have been proposed, see
section 2. Our resolving method based on sparse lifted edges would work with these error detectors as well and most likely yield better results due to more informative priors. We restrict ourselves to a simple false merge detector since the focus of this work is not the detector itself, but the framework of exploiting its predictions in an automated manner.Our classifier predicts whether the paths inside of segmented objects contain a false merge. We select all objects containing at least one path with the false merge probability larger than a preset threshold for resolving. For these objects, we add lifted edges between pairs of nodes corresponding to the terminals of the paths used for error prediction, deriving the weights from the path’s false merge probabilities.
Figure 1 shows an example false merger as a blue shaded object in (c) and the lifted edges derived from the corresponding paths in (d). We describe the experiment setup in more detail and show quantitative results in subsection 4.2.Sparse semantic labels
In this example, we tackle a segmentation problem in sponge choanocyte chambers. These structures are built from several surrounding cells, the choanocytes, that interact with the central cell via flagella which are surrounded by a collar of microvilli. Our goal is to segment cell bodies, flagella and microvilli. This task is challenging due to the large difference in sizes of these structures. Especially the segmentation of the small flagella and microvilli is difficult. Without the use of domain specific knowledge they are split into many pieces because of insufficient feature statistics due to small image boundaries.
In order to alleviate these false split errors, we automatically predict flagella and microvilli attribution. We then compute an instance segmentation based on these predictions by applying a threshold and connected components. We map the component labels to nodes of the graph, right column in Figure 1, panel(c). Then, we introduce attractive lifted edges between the nodes that were mapped to the same component label and repulsive lifted edges between nodes mapped to different component labels, see (d). Rather than deriving the lifted edge weights from data, we use preset values for attractiveness / repulsiveness. See subsection 4.3 for more details and results.
3.3 Hierarchical Lifted Multicut Solver
Finding the optimal solution of the lifted multicut objective is NPhard. Approximate solvers based on greedy algorithms [17] and fusion moves [3] have been introduced. However, even these approximations do not scale to the large problem we need to solve in the sponge segmentation example. In order to tackle this and even larger problems, we adapt the hierarchical multicut solver of [30] for lifted multicuts.
This solver extracts subproblems from a regular tiling of the volume, solves these subproblems in parallel and uses the solutions to contract nodes in the graph, thus reducing its size. This approach can be repeated for an increasing size of the blocks that are used to tile the volume, until the reduced problem becomes feasible with an other (approximate) solver.
We extend this approach to the lifted multicut by also extracting lifted edges during the subproblem extraction. We only extract lifted edges that connect nodes in the subgraph defined by the block at hand. This strategy, where we ignore lifted edges crossing block boundaries, is in line with the idea that lifted edges contribute to the energy, but not to the connectivity. Note that lifted edges that are not part of any subproblem at a given level will still be considered at a later stage. See algorithm 1 for pseudocode.
Note that this approach is conceptually similar to the fusion move based approximation of [4], which extracts and solves subproblems based on a random graph partition and accepts changes from subsolutions if they increase the overall energy, repeating this process until convergence. Compared to this approach, we extract subproblems from a deterministic partition of the graph. This allows us to solve only a preset number of subproblems leading to faster convergence.
Note that our approximate solver is only applicable if the graph at hand has a spatial embedding, which allows to extract subproblems from a tiling of space. In our case, this spatial embedding is given by the watershed fragments that correspond to nodes.
4 Results
We study the performance of the proposed method on three different problems: instance segmentation in murine cortex with priors from axon/dendrite segmentation, instance segmentation in drosophila brain with priors from morphologybased error detection and instance segmentation in a sponge choanocyte chamber with priors from semantic segmentation. See Table 3 for an overview of the different problem setups. We evaluate segmentation quality using the variation of information (VI) [25], which can be separated into split and merge scores, and the adapted rand score [2].
DISCLAIMER: THE RESULTS IN THIS SECTION SHOULD BE TREATED AS PRELIMINARY. WE ARE CURRENTLY TRAINING BETTER BOUNDARY DETECTORS WHICH WILL IMPROVE THE BASELINE SEGMENTATION QUALITY.
4.1 Axon Dendrite Attribution
We present results on a volume of murine somatosensory cortex that was acquired by FIBSEM at 5 5 6 nanometer resolution. The same volume has already been used in [18] for a similar experiment. To ensure a fair comparison between the two methods for incorporating axon/dendrite priors, we obtained derived data from the authors and use it to setup the segmentation problem.
This derived data includes probability maps for cell membrane, mitochondria, axon and dendrite attribution as well as a watershed oversegmentation derived from the cell membrane probabilities and groundtruth instance segmentation. From this data, we set up the graph partition problem as follows: we build the region adjacency graph from the watersheds and compute weights for the regular edges with a random forest based on edge and region appearance features. See [3] for a detailed description of the feature set. Next, we introduce dense lifted edges up to a graph distance of three. We use a random forest based on features derived from region appearance and clustering to predict their weights, see [3] for details. In addition to the region appearance features only based on raw data, we also take into account the mitochondria attribution here. Next, we map the axon/dendrite attribution to the nodes of and introduce sparse lifted edges between nodes mapped to different classes. We infer weights for these edges with a random forest based on features from the statistics of the axon and dendrite node mapping. We use the fusion move solver of [4] for optimizing the lifted multicut objective.
We divide the volume into a 1 3.5 3.5 micron block that is used to train the random forests for edge weights and a 2.5 3.5 3.5 micron block used for evaluation. The random forest predicting pixelwise probabilities was trained by the authors of [18] on a separate volume, using ilastik [34].
We compare the multicut and AMWC solutions reported in [18] with different variants of our methods, see Table 1. As a baseline, we compute the lifted multicut only with dense lifted edges and without features from mitochondria predictions (LMCD). We compute the full model with dense and sparse lifteds (LMCS) with and without additional features for dense lifted edges from mitochondria predictions. In addition, we compare to an iterative approach (LMCSI) similar to the error correction approach in subsection 4.2, where we perform LMCD segmentation first and introduce sparse lifted edges only for objects that contain a false merge (identified by presence of both axonic and dendritic nodes in the same object).
The LMCD segmentation quality is on par with the AMWC, although it does not use any input from the priors, showing the importance of dense lifted edges. Our full model with sparse lifted edges shows significantly better quality compared to LMCD. Mitochondriabased features provide a small additional boost. The segmentation quality of the iterative approach LMCSI is inferior to solving the full model LMCS. This shows the importance of joint optimization of the full model with dense and sparse lifted edges.
Method  VISplit  VIMerge  Rand Error 

MC [18]  0.3471  0.6347  0.0787 
AMWC [18]  0.4578  0.4935  0.0754 
LMCD  0.4144  0.4445  0.0891 
LMCS  0.4133  0.3788  0.0362 
LMCS (No Mitos)  0.4038  0.3966  0.0363 
LMCSI  0.5054  0.3998  0.0586 
4.2 False Merge Detector
We test the false merge correction on parts of the Drosophila medulla. This data was made public by the authors of [35], who also provide membrane probability maps, a watershed oversegmentation and a groundtruth segmentation for the whole volume.
In order to demonstrate segmentation improvement based on morphological knowledge, we train a random forest that can detect false merges in segmented objects. This classifier is based on morphological features derived from paths inside of segmented objects. These paths are extracted from skeletons computed with the method of [23] using the implementation from [36]. See section 7 for a detailed description of the feature set. The random forest is trained based on correct and falsely merged objects from an initial multicut segmentation of a separate part of the dataset.
Given this false merge detector, we set up the lifted multicut problem as follows: we compute the region adjacency graph based on the watersheds and compute weights for the regular edges with a random forest based on the edge and region appearance features described in [3]. We add dense lifted edges up to a graph neighborhood of three and compute weights based on region appearance and clustering, again following [3]. We solve this lifted multicut (LMCD) to obtain an initial segmentation and extract paths and features from it to apply the error detector. We introduce sparse lifted edges for all objects that contain at least one path with false merge probability larger than 0.5. Note that we introduce lifted edges corresponding to all paths in such an object and derive attractive and repulsive weights in order to improve the merge localisation. We then resolve these objects with a lifted multicut problem set up from regular edges as well as dense and sparse lifted edges (LMCSI). We use the fusion move solver of [4] to optimize the resulting lifted multicut objective.
We train the random forest for regular and dense lifted edges on a 2 2 2 micron cube and train the random forest for false merge detection on five 5 5 5 micron cubes cut out of the full volume. For inference, we cut out a 10 10
10 micron cube and split it into 8 subvolumes. We run inference on these subvolumes with LMCD and then resolve the objects predicted as false merges with with LMCSI. To estimate the upper limit of the performance of the proposed method, we run another experiment using an oracle predictor (always returning correct weights for the sparse lifted edges) instead of the error detection random forest. We refer to this approach as LMCSO. The results for all blocks are shown in
Figure 3. The raw numbers behind this figure can be found in Table 4, along with the Fscores of the false merge classifier. We observe a strong improvement of the VI merge scores for all blocks. Predictably, the number of false split errors goes up, but overall the LMCSI segmentation is better as shown by the improvement of the Rand Score in
Table 4. Figure 4 shows an example LMCSI result: the 5 objects visible in the figure were all merged into one by LMCD and resolved by LMCSI.4.3 Sponge Choanocyte Segmentation
The two previous experiments mostly profited from repulsive information derived from ultrastructure or morphology. In order to show how attractive information can be exploited, we turn to an instance segmentation problem in a sponge choanocyte chamber. The EM volume was imaged with FIBSEM at a resolution of 15 15 15 nanometer. We aim to segment structures of three different types: cell bodies, flagella and microvilli. Flagella and microvilli have a small diameter, which make them difficult to segment with a boundary based approach. On the other hand, cell bodies have a much larger diameter and touch each other, which makes a boundary based approach appropriate.
In order to setup the segmentation problem, we first compute probability maps for boundaries, microvilli and flagella attribution using the autocontext workflow of ilastik [34]. We setup the lifted multicut problem by first computing watersheds based on boundary maps, extracting the region adjacency graph and computing regular edge weights from the boundary maps accumulated over the edge pixels. We do not introduce dense lifted edges. For sparse lifted edges, we compute an additional instance segmentation of flagella and microvilli by thresholding the corresponding probability maps and running connected components. Then, we map the components of this segmentation to graph nodes and connect nodes mapped to the same component via attractive lifted edges and nodes mapped to different components via repulsive lifted edges. We use the hierarchical lifted multicut solver introduced in subsection 3.3 to solve the resulting objective, using the approximate solver of [17] to solve subproblems. Note that the full model contained too many variables to be optimized by any other solver in a reasonable amount of time.
We run our segmentation approach on the whole volume, which covers a volume of 70 75 50 microns, corresponding to 4600 5000 3300 voxels. For evaluation, we use three cutouts of size 15 15 1.5 microns with groundtruth for instance and semantic segmentation. We split the evaluation into separate scores for objects belonging to the three different structures, extracting them based on the semantic segmentation groundtruth. See Table 2 for the evaluation results, comparing the sparse lifted multicut (LMC) to the multicut baseline (MC). As expected the quality of the segmentation of cell bodies is not affected, because we don’t introduce lifted edges for those. The split rate in flagella and microvilli decreases significantly leading to a better overall segmentation for these structures.
Method  VISplit  VImerge  Rand Error 

Cells  
MC  0.6058  0.0116  0.0783 
LMC  0.6004  0.0116  0.0782 
Flagella  
MC  0.4728  0.0812  0.1205 
LMC  0.2855  0.0812  0.0429 
Microvilli  
MC  3.1760  1.1101  0.7409 
LMC  2.2745  1.1807  0.6973 
5 Discussion
We propose a general purpose strategy to leverage domainspecific knowledge for instance segmentation problems arising from EM image analysis. This strategy makes use of a graph partitioning problem known as lifted multicut by expressing the domain knowledge in the longrange lifted edges. We apply the proposed strategy to a diverse set of EM instance segmentation problems and consistently show an improvement in segmentation accuracy. For an application with ultrastructure based priors, we also observe that the lifted multicut based formulation yields higher quality results than the AMWC formulation of [18]. We believe that this is due to joint exploitation of dense shortrange and sparse longrange information. A complete joint solution, with both lifted edges and semantic labels, has recently been introduced in [24]. We look forward to exploring the potential of this objective for the neuron segmentation problem.
Similar to the findings of [21], we demonstrated that prevention of merge errors is more efficient than their correction: the joint solution of LMCS is more accurate than iterative LMCSI. However, not all prior information can be incorporated directly into the original segmentation problem. For these priors we demonstrate how to construct an additional resolving step which can also significantly reduce the number of false merge errors. In the future we plan to further improve our segmentations by other sources of information: matches of the segmented objects to known cell types, manual skeletons or correlative light microscopy imaging.
References
 [1] B. Andres et al. Globally optimal closedsurface segmentation for connectomics. In ECCV, 2012.
 [2] I. ArgandaCarreras, S. C. Turaga, D. R. Berger, D. Cireşan, A. Giusti, L. M. Gambardella, J. Schmidhuber, D. Laptev, S. Dwivedi, J. M. Buhmann, et al. Crowdsourcing the creation of image segmentation algorithms for connectomics. Frontiers in neuroanatomy, 9:142, 2015.
 [3] T. Beier et al. Multicut brings automated neurite segmentation closer to human performance. Nature Methods, 14(2), 2017.
 [4] T. Beier and others. An efficient fusion move algorithm for the minimum cost lifted multicut problem. In ECCV, 2016.
 [5] L. Breiman. Random forests. Machine learning, 45(1):5–32, 2001.
 [6] S. Chopra et al. The partition problem. Mathematical Programming, 59(13), 1993.
 [7] D. Ciresan, A. Giusti, L. M. Gambardella, and J. Schmidhuber. Deep neural networks segment neuronal membranes in electron microscopy images. In Advances in neural information processing systems, pages 2843–2851, 2012.
 [8] E. D. Demaine, D. Emanuel, A. Fiat, and N. Immorlica. Correlation clustering in general weighted graphs. Theoretical Computer Science, 361(23):172–187, 2006.
 [9] K. Dmitriev12, T. Parag, B. Matejek, A. E. Kaufman, and H. Pfister. Efficient correction for em connectomics with skeletal representation. In British Machine Vision Conferemce (BMVC), 2018.
 [10] A. Eberle, S. Mikula, R. Schalek, J. Lichtman, M. K. Tate, and D. Zeidler. Highresolution, highthroughput imaging with a multibeam scanning electron microscope. Journal of microscopy, 259(2):114–120, 2015.
 [11] K. Eichler et al. The complete connectome of a learning and memory centre in an insect brain. Nature, 548(7666), 2017.
 [12] J. Funke, F. D. Tschopp, W. Grisaitis, A. Sheridan, C. Singh, S. Saalfeld, and S. C. Turaga. Large scale image segmentation with structured loss based deep learning for connectome reconstruction. IEEE transactions on pattern analysis and machine intelligence, 2018.
 [13] K. J. Hayworth, C. S. Xu, Z. Lu, G. W. Knott, R. D. Fetter, J. C. Tapia, J. W. Lichtman, and H. F. Hess. Ultrastructurally smooth thick partitioning and volume stitching for largescale connectomics. Nature methods, 12(4):319, 2015.
 [14] L. Heinrich, J. Funke, C. Pape, J. NunezIglesias, and S. Saalfeld. Synaptic cleft segmentation in nonisotropic volume electron microscopy of the complete drosophila brain. In International Conference on Medical Image Computing and ComputerAssisted Intervention, pages 317–325. Springer, 2018.
 [15] A. Horňáková et al. Analysis and optimization of graph decompositions by lifted multicuts. In ICML, 2017.
 [16] M. Januszewski, J. Kornfeld, P. H. Li, A. Pope, T. Blakely, L. Lindsey, J. MaitinShepard, M. Tyka, W. Denk, and V. Jain. Highprecision automated reconstruction of neurons with floodfilling networks. Nature methods, 15(8):605, 2018.
 [17] M. Keuper et al. Efficient decomposition of image and mesh graphs by lifted multicuts. In ICCV, 2015.
 [18] N. Krasowski et al. Neuron segmentation with highlevel biological priors. IEEE transactions on medical imaging, 2017.
 [19] A. Kreshuk, U. Koethe, E. Pax, D. D. Bock, and F. A. Hamprecht. Automated detection of synapses in serial section transmission electron microscopy image stacks. PloS one, 9(2):e87351, 2014.
 [20] A. Kreshuk, C. N. Straehle, C. Sommer, U. Koethe, M. Cantoni, G. Knott, and F. A. Hamprecht. Automated detection and segmentation of synaptic contacts in nearly isotropic serial electron microscopy images. PloS one, 6(10):e24899, 2011.
 [21] T. Kroeger et al. Asymmetric cuts: Joint image labeling and partitioning. In GCVPR, 2014.
 [22] K. Lee et al. Superhuman accuracy on the snemi3d connectomics challenge. arXiv, 2017.
 [23] T.C. Lee et al. Building skeleton models via 3d medial surface axis thinning algorithms. CVGIP: Graphical Models and Image Processing, 1994.
 [24] E. Levinkov et al. Joint graph decomposition & node labeling: Problem, algorithms, applications. In CVPR, 2017.
 [25] M. Meilă. Comparing clusterings by the variation of information. In Learning theory and kernel machines. 2003.
 [26] Y. Meirovitch et al. A multipass approach to largescale connectomics. arXiv, 2016.
 [27] J. NixonAbell, C. J. Obara, A. V. Weigel, D. Li, W. R. Legant, C. S. Xu, H. A. Pasolli, K. Harvey, H. F. Hess, E. Betzig, et al. Increased spatiotemporal resolution reveals highly dynamic dense tubular matrices in the peripheral er. Science, 354(6311):aaf3928, 2016.
 [28] J. NunezIglesias, R. Kennedy, T. Parag, J. Shi, and D. B. Chklovskii. Machine learning of hierarchical clustering to segment 2d and 3d images. PloS one, 8(8):e71715, 2013.
 [29] S. Otsuka, A. M. Steyer, M. Schorb, J.K. Heriche, M. J. Hossain, S. Sethi, M. Kueblbeck, Y. Schwab, M. Beck, and J. Ellenberg. Postmitotic nuclear pore assembly proceeds by radial dilation of small membrane openings. Nature structural & molecular biology, 25(1):21, 2018.
 [30] C. Pape et al. Solving large multicut problems for connectomics via domain decomposition. In ICCV, 2017.
 [31] D. Rolnick et al. Morphological error detection in 3d segmentations. arXiv, 2017.
 [32] O. Ronneberger, P. Fischer, and T. Brox. Unet: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computerassisted intervention, pages 234–241. Springer, 2015.
 [33] M. R. Russell, T. R. Lerner, J. J. Burden, D. O. Nkwe, A. PelchenMatthews, M.C. Domart, J. Durgan, A. Weston, M. L. Jones, C. J. Peddie, et al. 3d correlative light and electron microscopy of cultured cells using serial blockface scanning electron microscopy. J Cell Sci, 130(1):278–291, 2017.
 [34] C. Sommer, C. Straehle, U. Koethe, and F. A. Hamprecht. Ilastik: Interactive learning and segmentation toolkit. In 2011 IEEE international symposium on biomedical imaging: From nano to macro, pages 230–233. IEEE, 2011.
 [35] S. Takemura et al. Synaptic circuits and their variations within different columns in the visual system of drosophila. PNAS, 112(44), 2015.
 [36] S. Van der Walt, J. L. Schönberger, J. NunezIglesias, F. Boulogne, J. D. Warner, N. Yager, E. Gouillart, and T. Yu. scikitimage: image processing in python. PeerJ, 2:e453, 2014.
 [37] J. G. White, E. Southgate, J. N. Thomson, and S. Brenner. The structure of the nervous system of the nematode caenorhabditis elegans. Philos Trans R Soc Lond B Biol Sci, 314(1165):1–340, 1986.
 [38] S. Wolf, A. Bailoni, C. Pape, N. Rahaman, A. Kreshuk, U. Köthe, and F. A. Hamprecht. The mutex watershed and its objective: Efficient, parameterfree image partitioning. arXiv preprint arXiv:1904.12654, 2019.
 [39] S. Wolf, C. Pape, A. Bailoni, N. Rahaman, A. Kreshuk, U. Kothe, and F. Hamprecht. The mutex watershed: efficient, parameterfree image partitioning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 546–562, 2018.
 [40] C. Xiao, W. Li, X. Chen, H. Han, and Q. Xie. An effective fully deep convolutional neural network for mitochondria segmentation based on atumsem. In Medical Imaging 2018: Image Processing, volume 10574, page 105740C. International Society for Optics and Photonics, 2018.
 [41] Z. Zheng, J. S. Lauritzen, E. Perlman, C. G. Robinson, M. Nichols, D. Milkie, O. Torrens, J. Price, C. B. Fisher, N. Sharifi, et al. A complete electron microscopy volume of the brain of adult drosophila melanogaster. Cell, 174(3):730–743, 2018.
 [42] J. Zung, I. Tartavull, K. Lee, and H. S. Seung. An error detection and correction framework for connectomics. In Advances in Neural Information Processing Systems, pages 6818–6829, 2017.
6 Appendix 1: Hierarchical Lifted Multciut Solver
7 Appendix 2: Skeleton Path Features
To extract features from skeleton paths, we take into account three different input maps: the raw data, the membrane probabilities and the distance transform to the boundary of the object from which the skelleton was extracted. We apply the following image filter on these inputs: Gaussian Smoothing, Laplacian of Gaussian, Hessian of Gaussian Eigenvalues. Each filter is applied with three different sigma values: 1.6, 4.2 and 8.4. To obtain features, we accumulate the values of these inputs over the skeleton path’s pixel and apply the following statistics: Mean, Variance, Skewness, Kurtosis, Quantiles.
8 Appendix 3: False Merge Correction Results
Block  1  2  3  4  5  6  7  8 

LMCD  
VIS  0.421  0.531  0.561  0.425  0.480  0.458  0.491  0.468 
VIM  0.766  0.996  0.797  0.745  0.753  0.818  0.853  0.812 
RAND  0.137  0.277  0.223  0.152  0.178  0.161  0.194  0.165 
LMCSI  
VIS  0.555  0.722  0.723  0.602  0.633  0.641  0.633  0.628 
VIM  0.598  0.752  0.651  0.572  0.581  0.646  0.694  0.630 
RAND  0.117  0.241  0.227  0.143  0.152  0.151  0.174  0.146 
LMCSO  
VIS  0.457  0.624  0.605  0.491  0.530  0.519  0.536  0.525 
VIM  0.494  0.642  0.548  0.480  0.489  0.518  0.532  0.523 
RAND  0.078  0.200  0.179  0.098  0.114  0.100  0.099  0.101 
We include additional material on the false merge correction results: the table underlying the results graphic, Table 4, evaluation curves for the random forest we have trained for false merge path classification Figure 5 and a more detailed figure of the example object shown in the results. Figure 6
FScore, accuracy, precision and recall curves for the 8 test blocks.