Frangi-Net: A Neural Network Approach to Vessel Segmentation

by   Weilin Fu, et al.

In this paper, we reformulate the conventional 2-D Frangi vesselness measure into a pre-weighted neural network ("Frangi-Net"), and illustrate that the Frangi-Net is equivalent to the original Frangi filter. Furthermore, we show that, as a neural network, Frangi-Net is trainable. We evaluate the proposed method on a set of 45 high resolution fundus images. After fine-tuning, we observe both qualitative and quantitative improvements in the segmentation quality compared to the original Frangi measure, with an increase up to 17% in F1 score.



There are no comments yet.


page 5


SVS-net: A Novel Semantic Segmentation Network in Optical Coherence Tomography Angiography Images

Automated vascular segmentation on optical coherence tomography angiogra...

Fine tuning U-Net for ultrasound image segmentation: which layers?

Fine-tuning a network which has been trained on a large dataset is an al...

EAR-U-Net: EfficientNet and attention-based residual U-Net for automatic liver segmentation in CT

Purpose: This paper proposes a new network framework called EAR-U-Net, w...

Fooling Neural Network Interpretations via Adversarial Model Manipulation

We ask whether the neural network interpretation methods can be fooled v...

Automated Pavement Crack Segmentation Using Fully Convolutional U-Net with a Pretrained ResNet-34 Encoder

Automated pavement crack segmentation is a challenging task because of i...

The Mertens Unrolled Network (MU-Net): A High Dynamic Range Fusion Neural Network for Through the Windshield Driver Recognition

Face recognition of vehicle occupants through windshields in unconstrain...

Semantic Segmentation of Thigh Muscle using 2.5D Deep Learning Network Trained with Limited Datasets

Purpose: We propose a 2.5D deep learning neural network (DLNN) to automa...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Fundus imaging can help to diagnose and monitor a number of diseases, such as diabetic retinopathy, glaucoma, age-related macular degeneration [1]. Visual analysis by a trained ophtamologist can be extremely time-consuming and hinder broad clinical application. To support this workflow, automatic segmentation [2] of the retinal vessel tree has been studies for decades, among which a vessel enhancement filter by Frangi et al. [3] is the most popular and forms the basis to various other strategies [4]

. However, this task is particularly challenging due to the complex structure of retinal vessels (e.g. branching, crossing) and low image quality (e.g. noise, artifacts, low resolution). In recent years, deep learning techniques have been exploited in the field of retinal vessel segmentation 

[5, 6]

. These approaches are data-driven, and tend to share a similar structure, where a classifier follows a feature extractor. For instance, Maji et al. 


use an auto-encoder as a feature extractor, and random forests as a classifier; Tetteh et al. 


apply a convolutional neural network (CNN) inception model for feature extraction and another CNN model for classification. These novel, data-driven approaches perform comparably to conventional methods, but do not yield significant improvements.

In this paper, we also investigate deep learning techniques on retinal vessel segmentation. To avoid an explicit separation into “feature extractor" and “classification", we propose an alternative approach, which formulates the Frangi filter as a pre-weighted network (“Frangi-Net"). The intuitive reasoning behind such an approach is that, by representing the Frangi filter as a pre-weighted neural network, subsequent training of the latter should improve segmentation quality. We aim to utilize prior knowledge of tube segmentation as a basis, and further improve it using a data-driven approach.

2 Materials and Methods

2.1 Frangi Vessel Enhancement Filter

The Frangi vesselness filter [3] is based on the eigen-value analysis of the Hessian matrix in multiple Gaussian scales. Coarse scale structures are typically obtained by smoothing the image with a Gaussian filter where

is the standard deviation. The Hessian matrix is a square matrix of second-order partial derivatives of the smoothed image. Therefore, it can be alternatively calculated by convolving the original image patch directly with a 2-D kernel

, which is the second-order partial derivatives of the Gaussian filter ,


The Hessian matrix is calculated as,


The two eigenvalues of the Hessian matrix are denoted by

and , which are calculated as


A high vesselness response is obtained if and satisfy the following conditions: , and . A mathematical description of the vesselness response is presented in [3],


where is the second order structureness, is the blobness measure, and stands for the vesselness value. are image-dependent parameters for blobness and structureness terms, and are set to 0.5, 1, respectively.

When using a Gaussian kernel with a standard deviation , vessels whose diameters equal to have the highest vesselness response. The diameter of retinal vessels in this work range from 6 to 35 pixels. Hence we choose a series of as pixels accordingly. Instead of convolving a patch with three different kernels (i. e., ), we create a three-level resolution hierarchy [4], and convolve each level with only. The resolution hierarchy consists of the original patch and two downsampled versions, using factors 2 and 4, respectively.

2.2 Pre-weighted Frangi-Net

Inspired by [7], we implement the multi-scale Frangi filter as a neural network called Frangi-Net on the basis of the previous section. The architecture of Frangi-Net is described in Fig. 1. The proposed method is used to analyze image patches, with an output size set to pixels. Input patches for the three sub-nets are cropped as in Fig. 1 (a).

For a single-scale sub-net, we first use a resize layer to downsample images to the corresponding size. Secondly, we use a 2-D convolution layer with three filters to get the Hessian matrix. These filters are initialized as second partial derivatives of the two-dimensional Gaussian function with . After that, a combination of mathematic operation layers are applied, based on Eqs. 3 and 4 to calculate eigen-values and vesselness responses of this scale.

(a) Patch Set
(b) Single-scale Net Structure
(c) Frangi-Net Overall Structure
Figure 1: The Frangi-Net architecture: In (a), dash line represents output patch size ; solid lines represent the patch set in the resolution hierarchy. (b) describes the dataflow through the framework of a single-scale net, where eigen-map 1 and 2 are maps of respectively; (c) describes the overall structure of Frangi-Net, where patch 0, patch 2, patch 4 correspond to size , , in (a).

The raw vesselness scores from Frangi-Net typically range from 0 to roughly 0.4. To create a binary vessel mask, a threshold is used, which is set to in this work. This means that most background pixels are squashed inbetween , and the rest few vessel pixels spread along

. To obtain a probability map, we subtract the data with the threshold

, and asymmetrically scale it, where negative values are multiplied with and positive values with so that raw scores are redistributed to . Afterwards, we apply a sigmoid layer. In this way, raw vesselness values of and are converted to and 1, respectively.

2.3 Trainable Frangi-Net

With the proposed network structure, we observe two sets of trainable parameters in the network: first, the covolution kernels responsible for the computation of the Hessian matrix; second, the parameters , that control the influence of structureness and blobness features. For each scale, this results in three trainable convolution kernels and two additional parameters that can be adapted during training. In Frangi-Net, single-scale nets are structured in parallel and updated independently, which means in total we have nine kernels and six parameters to update during training.

A high resolution fundus (HRF) image database [8] is used in this work. The database includes 15 images of each healthy, diabetic retinopathy, and glaucomatous subjects. The HRF images are high resolution pixel RGB fundus photographs. Only the green channels are used where vascular structures manifest a good contrast [9]. The intensity values are normalized to [0, 1]. We train Frangi-Net on each cartegory independently, randomly dividing the 15 datasets into 10 training, 2 validation and 3 testing sets. We also train the network with all 45 datasets, randomly taking 30 sets for training, 6 for validation and 9 for testing. As segmentation of retinal vessels from fundus images is an unbalanced classification problem, where the background pixels outnumber vessel pixels by approximately 10 to 1, dice coefficient loss [10] instead of the common cross entropy loss is used. The optimizer we utilize here is gradient descent with learning rate

and momentum 0.5. 1000 steps are used for training on each category independently, while 3000 steps are used for training on the whole database. Batch size is chosen as 250. We use python and tensorflow framework for the whole implementation.

3 Results

A quantitative comparison of the trained Frangi-Nets with the original Frangi filter is summarized in Table 1

. Each quality metric is calculated as an average of that of the testing datasets. After fine-tuning with four different training datasets, the F1 score, accuracy, precision and recall of the segmentation results on the testing datasets all get improved. Segmentation of heathy fundus images performs best, achieving a highest post-training F1 score as 0.712. Comparing the diseased cases to the healthy cases, the F1 score has a more significant improvement after training, up to


A comparison of the segmentation results before and after training using the healthy datasets is shown in Fig.2. Fig. 2(a) displays that Frangi-Net can segment main vessels well before training. However, it only partially segments the middle-sized vessels and fails to recognize most of the tiny ones. In contrast, after training, Frangi-Net is able to detect the middle-sized vessels well, and it also detects most of the tiny ones (Fig. 2(b)).

(a) Before
(b) After
Figure 2: Vessel segmentation results before and after training. The red, green and yellow colors represent the manual labels, the segmentation results and the overlaps between them, respectively.
Dataset F1 score accuracy precision recall
before after before after before after before after
Healthy 0.669 0.712 0.836 0.843 0.606 0.675 0.746 0.753
Diabetic retinopathy 0.495 0.532 0.819 0.822 0.524 0.588 0.468 0.486
Glaucomatous 0.495 0.584 0.838 0.841 0.477 0.562 0.608 0.618
Whole dataset 0.612 0.672 0.847 0.855 0.603 0.675 0.623 0.670
Table 1: Segmentation quality before and after training

4 Discussion

In this work, we proposed to combine the prior knowlegde about retinal vessel that is encoded in the Frangi-Filter with the data-driven capabilities of neural networks. We constructed a net that is equivalent to the multi-scale Frangi filter. We identified the trainable parts of Frangi-Net as convolutional kernels and parameters when computing vesselness. We redistributed the raw output vesselness values into a probability map and trained the net by optimizing dice coefficient loss. In experiments with high resolution fundus images of healthy and diseased patients, the trained Frangi-Net performs better than the original formulation both quantitatively and qualitatively.

Future work will investigate on different network architectures for Frangi-Net, or the combination of Frangi-Net with other networks. We will also evaluate Frangi-Net with other datasets.


  • [1] Maji D, Santara A, Mitra P, Sheet D. Ensemble of deep convolutional neural networks for learning to detect retinal vessels in fundus images. arXiv. 2016;.
  • [2] Kirbas C, Quek F. A review of vessel extraction techniques and algorithms. ACM Computing Surveys (CSUR). 2004;36(2):81–121.
  • [3] Frangi AF, Niessen WJ, Vincken KL, Viergever MA; Springer. Multiscale vessel enhancement filtering. MICCAI. 1998; p. 130–137.
  • [4] Budai A, Bock R, Maier A, Hornegger J, Michelson G. Robust vessel segmentation in fundus images. Int J Biomed Imaging. 2013;.
  • [5] Maji D, Santara A, Ghosh S, Sheet D, Mitra P; IEEE. Deep neural network and random forest hybrid architecture for learning to detect retinal vessels in fundus images. EMBC. 2015; p. 3029–3032.
  • [6] Tetteh G, Rempfler M, Zimmer C, Menze BH; Springer.

    Deep-FExt: Deep Feature Extraction for Vessel Segmentation and Centerline Prediction.

    International Workshop on Machine Learning in Medical Imaging. 2017; p. 344–352.

  • [7] Würfl T, Ghesu FC, Christlein V, Maier A; Springer. Deep learning computed tomography. Med Image Comput Comput Assist Interv. 2016; p. 432–440.
  • [8] Budai A, Odstrcilik J. High Resolution Fundus Image Database; 2013.
  • [9] Yin B, Li H, Sheng B, Hou X, Chen Y, Wu W, et al. Vessel extraction from non-fluorescein fundus images using orientation-aware detector. Med Image Anal. 2015;26(1):232–242.
  • [10] Taha AA, Hanbury A. Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Med Imaging. 2015;15(1):29.