Learning Hybrid Representations for Automatic 3D Vessel Centerline Extraction

12/14/2020 ∙ by Jiafa He, et al. ∙ Zhejiang University 10

Automatic blood vessel extraction from 3D medical images is crucial for vascular disease diagnoses. Existing methods based on convolutional neural networks (CNNs) may suffer from discontinuities of extracted vessels when segmenting such thin tubular structures from 3D images. We argue that preserving the continuity of extracted vessels requires to take into account the global geometry. However, 3D convolutions are computationally inefficient, which prohibits the 3D CNNs from sufficiently large receptive fields to capture the global cues in the entire image. In this work, we propose a hybrid representation learning approach to address this challenge. The main idea is to use CNNs to learn local appearances of vessels in image crops while using another point-cloud network to learn the global geometry of vessels in the entire image. In inference, the proposed approach extracts local segments of vessels using CNNs, classifies each segment based on global geometry using the point-cloud network, and finally connects all the segments that belong to the same vessel using the shortest-path algorithm. This combination results in an efficient, fully-automatic and template-free approach to centerline extraction from 3D images. We validate the proposed approach on CTA datasets and demonstrate its superior performance compared to both traditional and CNN-based baselines.



There are no comments yet.


page 3

page 7

page 9

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Extracting tubular objects, e.g., blood vessels, has become a crucial task in computer-assisted diagnosis (CAD) of many diseases. For example, vessel lumen segmentation and centerline extraction are prerequisites for vessel curved-planar reconstruction (CPR) [6] from computed tomography angiography (CTA) images, which further facilitates stenosis detection and plaque identification in clinical diagnosis. However, it is usually time-consuming to segment vessel and extract centerline from various medical images. Instead, automatic vessel segmentation and centerline detection play a more and more important role for quantitative analysis of vascular diseases. [1] [7] [26].

Recently, convolutional neural networks (CNNs) have been widely applied in 3D medical image segmentation. However, segmenting vessels in 3D medical images is still very challenging. The blood vessels have delicate tubular structures with a large variety in long-range topology, which cannot be captured by slice-wise or patch-wise convolutional operations in most deep learning based segmentation methods 

[5] [9] [24]. Moreover, at the presence of imaging artifacts which often exist in medical images, CNN-based segmentation algorithms are prone to missing some segments of vessels, resulting discontinuities in the extracted vessel centerline [18] [23] [28]. To preserve the correct topology of the extracted vessels, previous methods may rely on user input that annotates start and end points of each vessel [17] [22]. Then, a more complete vessel centerline can be found by a minimal-cost path-based algorithm [2] [4] [8] [10]. To avoid manual input, another approach to ensure correct topology is to build an atlas or template of the target vessels from training samples and register the template to the test image [27] [28]. However, this approach is not very generalizable as the vascular structure of the test sample might be very different from the template.

In this paper, we present a novel approach for vessel centerline extraction, which is able to ensure the connectivity of extracted vessels without the need of any manual input or vessel template. The key idea is to use a patch-wise 3D CNNs to segment vessel mask and regress vessel centerline heatmap from the input image, and meanwhile use another point-cloud network to label the extracted vessel segments, such that segments belonging to the same vessel can be connected in a post-processing step. This hybrid approach makes the best use of both worlds: patch-wise CNNs for local appearance learning and point-cloud networks for global geometry learning, resulting in a robust and efficient algorithm for automatic centerline extraction. We also propose a geometry-aware grouping strategy to improve the performance of point-cloud network for vessel labeling. The effectiveness of the proposed framework is validated on two datasets: a public dataset of coronary artery CTA scans and an in-house dataset of head and neck artery CTA scans. Experimental results show that our approach outperforms existing baseline methods in terms of both accuracy and completeness of extracted centerlines.

In summary, we make the following contributions: (1) A novel hybrid representation learning approach for fully-automatic and template-free vessel centerline extraction; (2) A geometry-aware grouping method that utilizes the skeleton’s connection property to improve the performance of vessel labeling; (3) The state-of-the-art performance on the public benchmark.

2 Methods

Given a 3D CTA image consisted of a sequence of 2D slices, the objective is to segment the arteries and delineate their centerlines. The state-of-the-art segmentation methods are mostly based on CNNs. Due to the heavy computation of 3D convolutions, an input 3D image needs to be divided into overlapped patches and fed into the segmentation network separately. This leads to restricted local receptive fields, which may not provide sufficient information to distinguish between arteries and veins, resulting false detection. Moreover, there is no guarantee on the connectivity of extracted vessel segments from patch-based CNNs. A solution is to connect the segments that belong to the same vascular branch in post-processing. But to achieve this we need to label all the segments, which is also difficult for a patch-based CNN as vessel labeling requires considering the global geometry of the vessels.

To address these issues, we propose a hybrid approach that consists of a patch-based CNNs for local vessel segmentation, a point-cloud network for global vessel labeling and a path-finding algorithm for final centerline extraction. Figure 1 provides an overview to our approach.

Figure 1: Overview of the proposed approach. The framework consists of three components: Vessel segmentation (blue block): the input CTA images are divided into 3D patches with overlap and then a 3D CNNs are utilized to efficiently learn local features of vascular objects. Vessel labeling (red block): vascular skeletons produced by thinning the segmentation results are fed into a point-cloud network to learn the global geometry of vessels and realize vascular branch labeling. Centerline extraction (green block): based on a cost map, which is constructed from the centerline heatmap and the labeled skeleton, a minimal cost path algorithm is finally utilized to extract the complete vessel centerline.

2.1 Vessel Segmentation

At first, we use a 3D CNNs to learn vascular local appearance features from 3D patches of the original CTA images and produce a coarse vessel segmentation. An UNet [15] backbone architecture with an encoding-decoding module is selected to transform the input image to the segmentation mask. Moreover, to explore long-range contextual information inside 3D patches, we embed a dual attention module [3] on top of the UNet backbone. Finally, a combination of a binary cross-entropy loss and a Dice loss is used as the total segmentation loss. Please refer to the supplementary for more details.

2.2 Vessel Labeling

Due to the lack of the global information in the patch-wise 3D CNNs, the vessel segments obtained from the vessel segmentation procedure tend to contain some false-positive results like veins and also miss some parts of tiny or tortuous vessels. We propose to perform a vessel labeling procedure that classifies the segmented vessels into different branches. Then such semantic labels can be used to remove non-vascular segments and group discontinuous vessel segments, as shown in Figure 3. The vessel labeling procedure is implemented by first generating a set of points which represent vascular skeleton from the vessel segmentation results and then using a point-cloud network to predict labels of these skeleton points.

Figure 2: Comparison. Patch-based 3D CNNs extract extra non-vascular tissues and miss some true segments (left). Our hybrid approach is able to remove non-vascular tissues and connect the disjointed segments (right).
Figure 3: Generated vessel skeletons. Head and neck arteries (the top row) are divided into 17 categories distinguished by color. Coronary artery (the bottom row) are divided into 3 categories including a category for false positives.

Point-cloud Generation. As it is inefficient to directly label vessel segments in 3D volumes using CNNs, we propose to perform the labeling on vascular skeletons represented by a set of points, which can reduce complexity and preserve original geometric information of vessels. To generate vascular skeletons from the vessel segments, a 3D thinning algorithm [11] is applied to erode the vessel segments and finally obtain single-voxel-width skeleton points, as shown in Figure 3. The generated skeletons are composed of discrete and unordered points lying on the center of the vascular lumen, which represent vascular geometry.

Vessel Labeling Network. Given a set of vascular skeleton points generated from vessel segments , the target of the vessel labeling network is to predict the label of each point, , as shown in Figure 3

. It is similar to point-cloud semantic segmentation tasks in 3D computer vision. Any point-cloud network can be adopted, such as the state-of-the-art PointNet++ 

[14] and dynamic graph CNNs (DGCNN) [21]. We will provide a comparison between them in experiments.

Figure 4: Comparison of points grouping methods. (a) Multi-scale grouping (MSG) in the PointNet++: the grouping areas are multi-scale -sphere and grouped points may come from different skeleton components. (b) Geometry-aware grouping (GAG): the grouping areas trend to stretch along the lines and group points from the same islands.
Figure 5: Centerline Extraction. (a) Centerline heatmap: the heatmap has larger value in point closer to the centerline and the points laying outside the vascular radius are set to a step-down low value. (b) Cost map: the cost map is built by combining the heatmap and the skeleton lines. (c) Minimal cost path: based on the cost map, a minimal cost path can be extracted to connect the broken lines as the complete vessel centerline.

Geometry-aware Grouping. A particular property of the skeleton points compared to a general point cloud is the given connectivity among adjacent skeleton points. Specifically, skeleton points can be divided into separated components using a connected-component labeling (CCL) algorithm [12]. However, both PointNet++ and DGCNN are realized to group points by -nearest neighbor (-NN) algorithm based on L2 distance, which ignores the given geometry of the vessel skeletons. As demonstrated in Figure 5(a), the L2-based methods are more likely to group points belonging to different components. To address this issue and leverage the skeletons’ connection property, we propose a geometry-aware grouping method (GAG) to modify the distance based on the connection relationship. The modified distance between two points is computed by


where is a weight, is the n-th component. As shown in Figure 5(b), the grouping area in GAG is prone to stretching along the skeleton lines and the points belonging to the same connected component are more likely to be grouped. This design facilitates local feature consistency in the same component and consequently improves the accuracy of vessel labeling as evaluated in the experiments.

2.3 Centerline Extraction

After vessel labeling, each vessel segment is assigned a semantic label. The semantic label can be used to remove non-vascular tissues and provide the guidance to connect disjointed vessel segments. Specifically, we first determine which segments should be connected and then use a minimal cost path method to connect them and output final centerlines based on a cost map. The cost map is constructed by a centerline heatmap regressed from another 3D CNNs and the labeled vessel skeletons.

Centerline Heatmap Regression. The centerline heatmap is defined as the opposite of a distance map, where points closer to the vessel centerline have larger values and points outside the vascular radius have a step-down low value, as shown in Figure 5

(a). Considering that the probability map obtained from the segmentation network has only learned the difference between background and the vessel (e.g. edge features) rather than the centerline feature within the vessel lumen, we adopt another 3D CNN similar to the network used in vessel segmentation to regress the centerline heatmap. The mean square error (MSE) loss is used to train the network. Based on the centerline heatmap, we construct a cost map to guide the minimal cost path search algorithm as shown in Figure

5(b). In areas where vessel segments exist, the cost map directly assigns a large value to the skeleton points (red lines in Figure 5(b)) and a small value to the points elsewhere (gray areas in Figure 5(b)).

Minimal Cost Path. Given vessel skeletons and their labels, we successively merge the disjointed segments with the same label. Paired boundary points with the same label are put in a priority queue according to the distance between the two points in the pair. We take a pair successively from the queue and then the minimal cost path is found by Dijkstra algorithm [20] to connect the two boundary points, as shown in Figure 5(c). If the two segments represented by the two points are connected in the previous step, we skip the pair and go on until the queue is empty.

Methods OV(%) OF(%) OT(%)
ModelDrivenCenterline [28] 92.4 80.6 93.4
SupervisedExtraction [7] 90.6 70.9 92.5
GFVCoronaryExtractor [23] 93.7 74.2 95.9
DepthFirstModelFit [25] 84.7 65.3 87.0
AutoCoronaryTree [19] 84.7 59.5 86.2
Our approach 95.8 72.3 96.3
Table 1: Performance comparison of automatic coronary artery centerline extraction methods. OV represents the completeness of extracted vessel centerline and is similar to Dice coefficient. OF determines the ratio of a coronary artery that has been extracted before making an error. OT gives an indication of how much the extracted centerline overlaps with the clinically relevant centerline reference (radius 0.75 mm).
Figure 6: Comparison of different grouping methods used in vessel labeling networks. It can be seen that the proposed GAG method achieves better local consistency of the predicted vessel labels.
Figure 7: Vessel curved-planar reconstruction (CPR) of the coronary artery based on the extracted centerline. Our approach can extract complete centerlines under serious imaging artifacts.
Vessel Labeling Accuracy(%) OV(%) OF(%) OT(%)
PointNet [13] 92.2 96.1 84.5 96.6
PointNet++ [14] 95.1 96.5 85.8 97.0
PointNet++(GAG) 97.0 97.2 85.7 97.6
DGCNN [21] 96.5 96.6 84.4 97.1
DGCNN(GAG) 97.2 97.0 84.9 97.4
Table 2: Performance comparison of different variants of vessel labeling networks on the private head and neck artery dataset. Four metrics are reported: the accuracy of vessel labeling and OV, OF and OT of four main vessel centerlines including left and right common carotid artery (L/RCCA), left and right vertebral artery (L/RVA).
Figure 8: Vessel segmentation, labeling and centerline extraction. Top: head and neck arteries. Note that the discontinuous RVA is connected in the centerline extraction procedure. Bottom: coronary arteries. Note that the non-vascular tissues are removed in the vessel labeling procedure and the discontinuous vessel segments are connected.

3 Experiments

We evaluate the proposed method on two datasets: a public coronary artery dataset and a private head and neck artery dataset. The first dataset is mainly used to compare our method with existing baseline methods in literature. The second dataset is mainly used for ablative study to verify our system designs. Following  [16], extracted centerlines are evaluated based on three metrics, namely total overlap (OV), overlap until first error (OF), and overlap with the clinically relevant part of the vessel (OT). The stage-wise results for the proposed framework are demonstrated in Figure 8.

Implementation details In vessel segmentation, we randomly crop 3D patches with the size of for the head and neck dataset and

for the coronary artery dataset. The ResNet34 is used as the encoder in the UNet architecture which starts with 32 feature channels that are doubled in each scale, and the max-pooling layer is removed from the original residual network. All convolutions are specified as

kernels, except the last two ResNet blocks, which are to reduce the parameter count. We employ the Adam optimizer with a polynomial learning rate which equals to

. In vessel labeling, the inputs are skeleton points, which are generated from vessel segments and resampled to 3000 points for each sample. The inital learning rate is set to 0.001, which is reduced by half every 30 epochs. The weight

in GAG is set to be 0.3.

Experiments on the Coronary Artery Dataset This public dataset contains 100 cardiac CT angiography (CCTA) scans collected from the clinic for training and 32 CCTA scans from  [16] for evaluation. We train the vessel segmentation network and vessel labeling network on the annotated coronary artery CTA images and the vessel skeletons are labeled as three categories including right arteries, left arteries and false-positive venous vessels. According to the ablation study in the head and neck artery dataset, we use the Pointnet++ with the GAG module as our vessel labeling network. The quantitative comparison is listed in Table 1 and the visualization of results is showed in Figure 7. Our hybrid approach achieves the highest performance in terms of OV and OT, respectively, indicating that the centerlines extracted by the proposed method are more complete than those produced by other methods.

Experiments on the Head and Neck Artery Dataset. This private dataset collected from the clinic contains 450 CTA scans, each of which has a manually annotated vessel mask. The dataset is split into 380 scans for training, 20 for validation and 50 for testing. In the dataset, vessel skeletons are labeled as 17 categories including left and right common carotid artery (L/RCCA), left and right vertebral artery (L/RVA), etc. Table 2 shows the evaluation results of several variants of our system with different point-cloud network designs. It can be seen that, with the geometry-aware grouping (GAG) method, the vessel labeling accuracy of both PointNet++ and DGCNN can be improved. Figure 7 shows the GAG can facilitate local consistency of the skeleton components.

4 Conclusions

We propose an automatic and template-free approach to 3D vessel centerline extraction based on hybrid representations, which ensures the connectivity of extracted centerlines. We show that the hybridization between learning local appearance with patch-based CNNs and learning global geometry with point-cloud networks results in an efficient and robust framework to extract geometric objects from 3D data. We demonstrate superior performance on artery centerline extraction from CTA images and believe that the proposed approach can also be applied in other centerline or skeleton extraction tasks.

4.0.1 Acknowledgements

This work is funded by National Key Research and Development Program of China (No. 2019YFC0118100), and is partially supported by National Key Research and Development Program of China with Grant No. 2018AAA0101900/2018AAA0101902, Beijing Municipal Commission of Science and Technology under Grant No. Z181100008918005, the National Natural Science Foundation of China (NSFC Grant No. 61772039 and No. 91646202), and Hong Kong Research Grant Council [12301417, 16307818, 16301419]; Hong Kong University of Science and Technology [R9405, IGN17SC02, Z0428].


  • [1] S. Cetin and G. Unal (2015)

    A higher-order tensor vessel tractography for segmentation of vascular structures

    IEEE transactions on medical imaging 34 (10), pp. 2172–2185. Cited by: §1.
  • [2] T. Deschamps and L. D. Cohen (2001) Fast extraction of minimal paths in 3d images and applications to virtual endoscopy. Medical image analysis 5 (4), pp. 281–299. Cited by: §1.
  • [3] J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu (2019) Dual attention network for scene segmentation. In CVPR, Cited by: §2.1.
  • [4] M. A. Gülsün, G. Funka-Lea, P. Sharma, S. Rapaka, and Y. Zheng (2016) Coronary centerline extraction via optimal flow paths and cnn path pruning. In Medical Image Computing and Computer-Assisted Intervention - MICCAI 2016, S. Ourselin, L. Joskowicz, M. R. Sabuncu, G. Unal, and W. Wells (Eds.), pp. 317–325. External Links: Document Cited by: §1.
  • [5] K. Kamnitsas, C. Ledig, V. F. Newcombe, J. P. Simpson, A. D. Kane, D. K. Menon, D. Rueckert, and B. Glocker (2017) Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation. Medical image analysis 36, pp. 61–78. Cited by: §1.
  • [6] A. Kanitsar, D. Fleischmann, R. Wegenkittl, P. Felkel, and M. E. Gröller (2002) CPR: curved planar reformation. In VIS, Cited by: §1.
  • [7] Y. Kitamura, Y. Li, and W. Ito (2012) Automatic coronary extraction by supervised detection and shape matching. In ISBI, Cited by: §1, Table 1.
  • [8] H. Li and A. Yezzi (2007) Vessels as 4-d curves: global minimal 4-d paths to extract 3-d tubular surfaces and centerlines. IEEE transactions on medical imaging 26 (9), pp. 1213–1223. Cited by: §1.
  • [9] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. Van Der Laak, B. Van Ginneken, and C. I. Sánchez (2017) A survey on deep learning in medical image analysis. Medical image analysis 42, pp. 60–88. Cited by: §1.
  • [10] S. Moriconi, M. A. Zuluaga, H. R. Jäger, P. Nachev, S. Ourselin, and M. J. Cardoso (2017) VTrails: inferring vessels with geodesic connectivity trees. In International Conference on Information Processing in Medical Imaging, pp. 672–684. Cited by: §1.
  • [11] K. Palágyi, E. Balogh, A. Kuba, C. Halmai, B. Erdőhelyi, E. Sorantin, and K. Hausegger (2001) A sequential 3d thinning algorithm and its medical applications. In IPMI, Cited by: §2.2.
  • [12] J. Park, C. G. Looney, and H. Chen (2000) Fast connected component labeling algorithm using a divide and conquer technique.. Computers and Their Applications 4, pp. 4–7. Cited by: §2.2.
  • [13] C. R. Qi, H. Su, K. Mo, and L. J. Guibas (2017) Pointnet: deep learning on point sets for 3d classification and segmentation. In CVPR, Cited by: Table 2.
  • [14] C. R. Qi, L. Yi, H. Su, and L. J. Guibas (2017) Pointnet++: deep hierarchical feature learning on point sets in a metric space. In NIPS, Cited by: §2.2, Table 2.
  • [15] O. Ronneberger, P. Fischer, and T. Brox (2015) U-net: convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi (Eds.), pp. 234–241. External Links: Document Cited by: §2.1.
  • [16] M. Schaap, C. T. Metz, T. van Walsum, A. G. van der Giessen, A. C. Weustink, N. R. Mollet, C. Bauer, H. Bogunović, C. Castro, X. Deng, et al. (2009) Standardized evaluation methodology and reference database for evaluating coronary artery centerline extraction algorithms. Medical image analysis 13 (5), pp. 701–714. Cited by: §3, §3.
  • [17] R. Selvan, J. Petersen, J. H. Pedersen, and M. de Bruijne (2018) Extracting tree-structures in ct data by tracking multiple statistically ranked hypotheses. arXiv preprint arXiv:1806.08981. Cited by: §1.
  • [18] R. M. Stefancik and M. Sonka (2001) Highly automated segmentation of arterial and venous trees from three-dimensional magnetic resonance angiography (mra). The international journal of cardiovascular imaging 17 (1), pp. 37–47. Cited by: §1.
  • [19] H. Tek, M. A. Gulsun, S. Laguitton, L. Grady, D. Lesage, and G. Funka-Lea (2008) Automatic coronary tree modeling. The Insight Journal. Cited by: Table 1.
  • [20] L. Verscheure, L. Peyrodie, N. Makni, N. Betrouni, S. Maouche, and M. Vermandel (2010) Dijkstra’s algorithm applied to 3d skeletonization of the brain vascular tree: evaluation and application to symbolic. In EMBC, Cited by: §2.3.
  • [21] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon (2018) Dynamic graph cnn for learning on point clouds. arXiv preprint arXiv:1801.07829. Cited by: §2.2, Table 2.
  • [22] J. M. Wolterink, R. W. van Hamersvelt, M. A. Viergever, T. Leiner, and I. Išgum (2019) Coronary artery centerline extraction in cardiac ct angiography using a cnn-based orientation classifier. Medical image analysis 51, pp. 46–60. Cited by: §1.
  • [23] G. Yang, P. Kitslaar, M. Frenay, A. Broersen, M. J. Boogers, J. J. Bax, J. H. Reiber, and J. Dijkstra (2012) Automatic centerline extraction of coronary arteries in coronary computed tomographic angiography. The international journal of cardiovascular imaging 28 (4), pp. 921–933. Cited by: §1, Table 1.
  • [24] Q. Yu, L. Xie, Y. Wang, Y. Zhou, E. K. Fishman, and A. L. Yuille (2018)

    Recurrent saliency transformation network: incorporating multi-stage visual cues for small organ segmentation

    In CVPR, Cited by: §1.
  • [25] S. Zambal, J. Hladuvka, A. Kanitsar, and K. Bühler (2008) Shape and appearance models for automatic coronary artery tracking. the midas journal. In MICCAI Workshop–Grand Challenge Coronary Artery Tracking. http://hdl.handle.net/10380/1420, Cited by: Table 1.
  • [26] Z. Zhai, M. Staring, I. Hernández Girón, W. J. Veldkamp, L. J. Kroft, M. K. Ninaber, and B. C. Stoel (2019) Automatic quantitative analysis of pulmonary vascular morphology in ct images. Medical physics 46 (9), pp. 3985–3997. Cited by: §1.
  • [27] D. P. Zhang, L. Risser, F. Vialard, P. Edwards, C. Metz, L. Neefjes, N. Mollet, W. Niessen, and D. Rueckert (2010)

    Coronary motion estimation from cta using probability atlas and diffeomorphic registration

    In MIVR, Cited by: §1.
  • [28] Y. Zheng, H. Tek, and G. Funka-Lea (2013) Robust and accurate coronary artery centerline extraction in cta by combining model-driven and data-driven approaches. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2013, K. Mori, I. Sakuma, Y. Sato, C. Barillot, and N. Navab (Eds.), pp. 74–81. External Links: Document Cited by: §1, Table 1.