The Generation and Application of Medical Image Grid Based on Differential Geometric Features

01/30/2019 ∙ by Yongpei Zhu, et al. ∙ Tsinghua University 0

Accurate segmentation of brain tissue in magnetic resonance images (MRI) is a difficult task due to different types of brain abnormalities. In this paper, we review the deformation method focus on the construction of diffeomorphisms, address clearly a new formation of the deformation problem for moving domains, and we apply it in natural images, face images and MRI brain images. And we use a new method to construct diffeomorphisms through a completely different approach. The idea is to control directly the Jacobian determinant and the curl vector of a transformation and use them as one CNN channel with other modalities(T1-weighted, T1-IR and T2-FLAIR) to get more accurate results of brain segmentation. More importantly, we discuss the influence of some optimization parameters to precision analysis of MRI brain segmentation by both numerical experiments and theoretical analysis. We test this method on the IBSR dataset and MRBrainS18 dataset based on VoxResNet and prove the influence of three parameters on the accuracy of MRI brain segmentation.Finally, we also compare the segmentation performance of our method in two networks, VoxResNet and 3D U-Net network. We believe the proposed method can advance the performance in brain segmentation and clinical diagnosis.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 6

page 8

page 9

page 10

page 11

page 12

page 14

page 15

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Magnetic resonance imaging (MRI) is usually the preferred method of structural brain analysis, as it provides high-contrast and high-spatial resolution images of soft tissue with no known health risks, which is the most popular choice to analyze the brain and we will focus on MRI in this work. Quantitative analysis of brain MR images is routine for many neurological diseases and conditions. Segmentation, i.e., labeling of pixels in 2D (voxels in 3D), is a critical component of quantitative analysis. Brain tissue segmentation generally refers to the separation of the brain into three functional components, namely, cerebrospinal fluid (CSF), grey matter (GM) and white matter (WM). There is a need for automated segmentation methods to provide accuracy close to manual segmentation with a high consistency.

Deep learning techniques are gaining popularity in many areas of medical image analysis [12]

, different from traditional machine learning algorithm, it is a new and popular machine learning technique, which can extract complex feature levels from images. Some of the known deep learning algorithms are stacked auto-encoders, deep Boltzmann machines, deep neural networks, and convolutional neural networks (CNNs). CNNs are the most commonly applied to image segmentation and classification.


Diffeomorphism is an active research topic in differential geometry. J.Moser first proved the existence of diffeomorphism under a Jacobian determinant constraint. Later, Dr.Liao proposed the deformation method to construct diffeomorphisms. A div-curl system is created in the construction of diffeomorphisms. Since the Jacobian determinant has a direct physical meaning in grid generation which represents the size of grid cell, and the curl-vector represents the grid cell rotations, the deformation method was applied successfully to grid generation and adaptation problems.

Nowadays, the application of differential geometry in deep learning is more and more extensive, especially in the field of medical image[5]

. Based on the manifold characteristics of differential geometry, the success of deep learning is attributed to the inherent laws of the data itself. High-dimensional data are distributed near low-dimensional manifolds, which have a specific probability distribution, and it is also attributed to the strong ability of deep learning network to approximate nonlinear mapping. Deep learning technology can extract manifold structure from a kind of data and express the global prior knowledge with manifold, specifically, encoding and decoding mapping, which is implied in the weight of neurons. In the field of medical image analysis, doctors can determine whether the organs are abnormal by precisely comparing the geometry of the organs. By analyzing the geometric features of the tumor, we can judge the benign and malignant nature of the tumor. It can be attributed to the registration and analysis of medical images

[Dalca2018Unsupervised]. The deep learning method based on differential geometry plays an important role in medical image registration.

In this paper, we review the deformation method [12], focus on the construction of diffeomorphisms, address clearly a new formation of the deformation problem for moving domains, and we apply it in natural images, face images and MRI brain images. In theory, the deformation method provides one diffeomorphic solution to a nonlinear differential equation. Based on calculus of variation and optimization, we proposed a new variational method with prescribed Jacobian determinant and curl vector and use them as one CNN channel with other modalities to get more accurate results of brain segmentation. More importantly, we observed the important role of Jacobian determinant and the curl vector in determining a diffeomorphism and hence we discuss the influence of some optimization parameters to precision analysis of MRI brain segmentation by both numerical experiments and theoretical analysis.We test this method on the IBSR dataset and MRBrainS18 dataset based on VoxResNet and prove the influence of three parameters on the accuracy of MRI brain segmentation.

2 Related Work

In differential geometry, a diffeomorphism is map between manifolds which is differentiable and has a differentiable inverse, and it is a active research topic. In 1990, J.Moser proved the existence of diffeomorphisms such that

(1)

Since 1992-1993, G.liao [Liao2016A] improved J.Moser’s method and proposed the deformation method of grid generation.In this method, a diffeomorphism is numerically constructed, such that maps a grid of to a new grid of , whose grid sizes(approximated by the Jacobian determinant of ,namely equal to a prescribed scalar monitor function at . Correspondingly, the formula (1) is extended to the following one which is more meaningful in the physical domain:

(2)

This is known as the deformation method. A series of applications were made including adaptive moving grid [11], and steady euler flow calculations [13].In addition,the dynamic version of the deformation method on fixed domains was developed in [11] based on solving Poisson’s equations. In 2004,The Least-Squares Finite Element Method was first applied to solve the div-curl system in [Fleitas2005The] which extends the deformation method of grid deneration to moving domains.This version constructs a diffeomorphism , for

(3)

The success of the deformation method of grid generation actually relies on the div-curl system, which shows the great importance of the curl vector (). Hence, in the paper of 2016 [5], Xi Chen proposed another completely new approach by directly applying JD(the Jacobian determinant ) and CV(the curl vector).They used calculus of variation to formulate a new variational method with prescribed JD and CV of constructing transformations [5]. A recovering experiment was designed by the new variational method with the same JD and CV.

Accurate automated segmentation of brain structures such as white matter (WM), gray matter (GM), and cerebrospinal fluid (CSF) in MRI is important for studying early brain development in infants and precise assessment of the brain tissue. The application of differential geometry in deep learning is more and more extensive, especially in the field of medical image. In recent years, CNNs have been used in the segmentation of brain tissue, avoiding a clear definition of spatial and strength characteristics and providing better performance than the classical approach.

In this paper, we apply the deformation method in natural images, face images and MRI brain images. Based on calculus of variation and optimization, we proposed a new variational method with prescribed Jacobian determinant and curl vector and use them as one CNN channel with other modalities to get more accurate results of brain segmentation. More importantly, we discuss the influence of some optimization parameters to precision analysis of MRI brain segmentation by both numerical experiments and theoretical analysis. We test this method on the IBSR dataset and MRBrainS18 dataset based on VoxResNet and prove the influence of three parameters on the accuracy of MRI brain segmentation.

3 Method

3.1 Deformation Method

As a competitive method in solving grid generation problems, the deformation method of grid generation usually focuses on constructing one diffeomorphism.For given monitor function ,let on and apply the deformation method to construct such that .().Also there are some boundary conditions.Indeed, the deformation method is a method to construct with property for given monitor function . The deformation method [13],[11],[5] is derived from differential geometry. Consider and (two and three dimension) with , be moving (includes fixed) domains. Let be the velocity field on , where on any part of with slippery-wall boundary conditions where is the outward normal vector of . Given diffeomorphism and scalar function on the domain , such that

(4)

A new (differ from ) diffeomorphism , such that , , can be constructed the following two steps:

  • First, determine on by solving

    (5)
  • Second, determine on by solving

    (6)

For computational simplicity system (5) is modified into a Poisson equation as follows. Let , then

(7)

Hence, determination of is depended on , which is cheaper to find.

3.1.1 Algorithm Implementation

The deformation method has implemented in 4 steps.

  • Step 1:Initialize.

    Starts from

  • Step 2:Div-curl system(5).

    Compute then use LSFEM to solve (5),get on Actually, we apply Finite Difference Method(FDM) instead of LSFEM in the implementation.

  • Step 3:ODE(6).

    Update from to by (6).

  • Step 4:Next time step.

    Move to next time step ,back to Step 2 till t=1.

3.1.2 Precision Analysis

When using numerical methods to implement the above algorithm of constructing ,the precision relies on the following parts:

  • Parameter 1:dt

    Evaluating with a Poisson equation, with a optimization parameter .

  • Parameter 2:h

    Computing the gradient when solving the div-curl system(JD and CV)(5),with a optimization parameter h, which set h as the spacing between points in each direction.

  • Parameter 3:N

    Computing JD and CV by generating grid cells(5), with a optimization parameter N, which set N as the size of grid notes.

3.2 Variational Method with JD and CV

In the deformation method,we first determine a velocity field , then use it to construct .Inspired from the div-curl system, with the same Jacobian determinant and different curl vectors can lead different diffeomorphisms [5].The curl vector of is also needed in the cost function based on the new approach to new variational method with prescribed JD and CV.Also, numerical construction of diffeomorphisms.

Given a domain , and a scalar function defined on with

(8)

Define a cost functional ssd to be:

(9)

Here is a weight parameter, is the prescribed Jacobian determinant monitor function, and is the prescribed curl vector monitor function.

  • Removed the artificial time step, directly let

    (10)
  • The constraint were generalized to

    (11)

    Which leads to(in 2D)

    (12)

    We then used and as the control function and directly set:

    (13)

Combining(9)(10) and(13), the described is named new variational method with prescribed Jacobian determinant and curl vector.

3.2.1 Algorithm Implementation

We can implement the new version by similar gradient descent optimization scheme as follows:

  • Step 1:Initialize

  • Step 2:Compute ,then solve Poisson’s equation to get .

  • Step 3:Update f by ,where tstep is an optimization parameter.

  • Step 4:Solve Poisson’s equation to get .

  • Step 5:Update by (10).

  • Step 6:Back to step 2,keep iterating until a preset tolerance or a preset number of iteration steps is reached.

3.3 Experiments of the Deformation Method

Here we demonstrate some simulations of the deformation method. With both the Jacobian and curl term(div-curl system) here in our algorithm, we can construct a transformation, which proves the algorithm is reliable and accurate in grid generation.

Example one:Natural images

We design a constructing experiment to test the accuracy of our method and discover more details insides with natural images. We give the grid image of the map of (a)Bridge, (b)Car and (c)Pyramid. All images are given a one diffeomorphism from the square(

)(phi1 and phi2).From the original natural images and grid cells,we can see that the diffeomorphism mapping preserves the essential manifold features of the original image which may be used in the feature extraction of CNN in deep learning.

Figure 1: Original natural images as of (4): (a) Image: Bridge,(b) Image: Car (c) Image: Pyramid.
Figure 2: Grids generated by setting intensity of natural images(size ) as of (4): (a) Grid: Bridge,(b) Grid: Car (c) Grid: Pyramid.

Example two:Face images

We design a constructing experiment to test the deformation method and discover more details insides with human’s face images. We give the grid image of the map of three female faces. All images are given a one diffeomorphism from the square()(phi1 and phi2), we want to construct

from div-curl system. We can see that the grid image provide the manifold features which can be used in face recognition and detection.

Figure 3: Grid generated by face image(size ): (a) First face, (b) Second face, (c) Third face.

Example three:MRI brain images

We also design a constructing experiment to the diffeomorphism of brain MRI images and display them in three dimensions. Given a nonlinear diffeomorphism from the cube (phi1, phi2 and phi3) to itself for different space of grid notes(h=16,24 and 32), we want to construct from its jacobian determinant and curl(div-curl system). We show the grid image from full scale view, front view, lateral view, vertical view respectively when the grid space get the value 6.

Figure 4: Full scale view of grid generated by MRI image(size ): (a)Grid space=16, (b)Grid space=24, (c)Grid space=32.
Figure 5: Grid generated by MRI image(size ,the grid space is 6): (a)Front view, (b)Lateral view, (c)Vertical view.

3.4 Precision Analysis Experiments of the Deformation Method

Example one:Parameter N for Precision Analysis

In this experiment, we find that parameter N(the size of grid notes) has a influence on the precision of grid generation when constructing the diffeomorphism .We set N as 64, 128 and 256 and make comparison between them.We find the depth of the grid color reflects the density between the grids.With the increase of the grid size, the diffeomorphism becomes denser and the grid color is deeper, indicating that the generated mesh can more accurately express the details of the original image.This can be used in improving the accuracy of MRI segmentation when using them as one CNN channel by increasing the parameter N.

Figure 6: Original MRI image: (a) First slice, (b) Second slice, (c) Third slice.
Figure 7: Grid generated by MRI image(size ): (a) First slice, (b) Second slice, (c) Third slice.
Figure 8: Grid generated by MRI image(size ): (a) First slice, (b) Second slice, (c) Third slice.
Figure 9: Grid generated by MRI image(size ): (a) First slice, (b) Second slice, (c) Third slice.

Example two:Parameter dt for Precision Analysis

When evaluating with the Poisson equation,we find the parameter has influenced the accuracy of grid generation.we set as 1,5,10,50 and 100( Corresponds to 1,0.2,0.1,0.02 and 0.01 respectively). We can see that the morphology of diffeomorphism is more close to the original image with the increase of (correspond to the decrease of ), especially when is 1 and 5, indicating that the generated grid is more accurate in expressing the details of the original image. However, as the size continues to increase, the large details are very close, and the difference lies in some small details,especially when reach the value 10,50 and 100.

Figure 10: Grid generated by MRI image(dt=1): (a) First slice, (b) Second slice, (c) Third slice.
Figure 11: Grid generated by MRI image(dt=0.2): (a) First slice, (b) Second slice, (c) Third slice.
Figure 12: Grid generated by MRI image(dt=0.1): (a) First slice, (b) Second slice, (c) Third slice.
Figure 13: Grid generated by MRI image(dt=0.02): (a) First slice, (b) Second slice, (c) Third slice.
Figure 14: Grid generated by MRI image(dt=0.01): (a) First slice, (b) Second slice, (c) Third slice.

Example three:Parameter h for Precision Analysis

When Computing the gradient when solving the div-curl system(JD and CV)(5),the parameter h also has an impact on the precision analysis of grid generation, which set h as the spacing between points in each direction.We set h as 1,3,5 and 10 respectively and the space between the grids becomes more sparse, and the details become more fuzzy and unable to approximate the original image with the increase of h.

Figure 15: Grid generated by MRI image(h=1): (a) First slice, (b) Second slice, (c) Third slice.
Figure 16: Grid generated by MRI image(h=3): (a) First slice, (b) Second slice, (c) Third slice.
Figure 17: Grid generated by MRI image(h=5): (a) First slice, (b) Second slice, (c) Third slice.
Figure 18: Grid generated by MRI image(h=10): (a) First slice, (b) Second slice, (c) Third slice.

3.5 Recovering Experiments of Variational Method Based on JD and CV

Based on calculus of variation and optimization, we used variational method generate the prescribed Jacobian determinant and curl vector of MRI brain images.We use MATLAB to generate the grid images based on JD and CV of brain MRI images as above.And we extract the images formed by JD and CV information from the grid images above, and saved it as nii image format. The following figures show in order are the JD and CV images of three slices, which present geometric deformation features of the image, especially highlight the change of morphological features of CSF, GM, WM.

Figure 19: Two-dimensional generated images based on JD: (a) The image formed by JD of first slice, (b) The image formed by JD of second slice, (c) The image formed by JD of third slice.
Figure 20: Two-dimensional generated images based on CV: (a) The image formed by CV of first slice, (b) The image formed by CV of second slice, (c) The image formed by CV of third slice.

3.6 VoxResNet Architecture

We test this method on the datasets based on the deep voxelwise residual network VoxResNet which shows in [Hao2016VoxResNet]

. Deep residual networks with residual units have shown great accuracy on several image recognition tasks such as ImageNet

[9] [10] and MS COCO [6] competitions. Generally, deep residual networks(ResNets)[9] consist of many ”Residual Units” and the residual unit can be expressed as following:

(14)

here the denotes the residual function, is the input feature to the residual unit and denotes the desired underlying identity mapping. As shown in [Hao2016VoxResNet], the VoxResNet consists of stacked residual modules with a total of 25 volumetric convolutional layers and 4 deconvolutional layers [14]. In each residual unit, the input and transformed feature are added together with skip connection, hence the information can be directly propagated in the forward and backward passes and it is implemented in a 3D way to strengthen the volumetric feature representation.

3.7 Multi-modalities

In medical image analysis, 3D volumetric data is usually obtained in a variety of ways for robust detection of different organizational structures. For example, three modalities including T1, T1-weighted inversion recovery (T1-IR) and T2-FLAIR are usually available in brain structure segmentation task [16]. T1 image has good anatomical structure and T2 image can show good tissue lesions, T1-IR image has strong T1 contrast characteristics, and T2-FLAIR is often used to inhibit CSF issues. The main reason for obtaining multimodal images is that the information of multimodal data sets can supplement each other and provide robust diagnostic results. Thus, we concatenate these multi-modality data with JD and CV data as input, then in the process of network training, complementary information is combined and fused in an implicit way, which is more consistent than any single training method. The following figure shows each modality of brain image and the image formed by its JD and CV information.

Figure 21: Each modality of brain image and the image formed by its JD and CV information: (a) T1 without skull stripping, (b) T1-IR without skull stripping, (c) T2-FLAIR without skull stripping, (d) T1 with skull stripping (e) The image formed by JD of T1, (f) The image formed by CV of T1.

3.8 Proposed Framework

Figure 22 shows the process framework of our proposed method. All modalities should be skull stripped and we can extract differential geometric features including the Jacobian determinant(JD) and the curl vector(CV) derived from T1 modality. After all modalities, including image labels(ground truth) have preprocessed, three modalities including T1-weighted, T1-IR and T2-FLAIR images with JD or CV image will be concatenated together as a new multi-modality. And it will be used as input of VoxResNet network. After training and testing, the predicted result with Ground Truth will be calculated for each tissue type(CSF, GM and WM), respectively, including DSC (Dice coefficient), HD ( Hausdorff distance) and AVD (Absolute Volume Difference) as evaluation criteria of segmentation. We will perform all the experiments based on this process.

Figure 22: The process framework of our proposed method.

4 Results

4.1 Datasets and Pre-processing

4.1.1 Datasets

The Internet Brain Segmentation Repository (IBSR) provides manually-guided expert segmentation results along with magnetic resonance brain image data. Its purpose is to encourage the evaluation and development of segmentation methods. The dataset consists of 18 MRI volumes and the corresponding ground truth (GT) is provided. The data can be downloaded from 111https://www.nitrc.org/frs/?group_id=48.

The aim of MRBrainS challenge is to segment brain into four-class structures, namely background, cerebrospinal fluid (CSF), gray matter (GM) and white matter (WM). Multi-sequence 3T MRI brain scans, including T1-weighted, T1-IR and T2-FLAIR are provided for each subject. Five brain MRI scans with manual segmentations are provided for training and 15 only MRI scans are provided for testing. The data can be downloaded from 222http://mrbrains18.isi.uu.nl/ for MRBrainS18 dataset and 333http://mrbrains13.isi.uu.nl/ for MRBrainS13 dataset.

4.1.2 Image Preprocessing

Typical preprocessing steps for structural brain MRI include the following key steps [1]

: registration, skull stripping, bias field correction, intensity normalization and noise reduction. All the datasets we use are already skull stripped. In this paper, we subtract Gaussian smoothed image, and apply intensity normalization(z-scores) and Contrast-Limited Adaptive Histogram Equalization (CLAHE) for enhancing local contrast by

[21] in the pre-processing step. Then multiple input volumes pre-processed were used as input data in our experiments.

4.2 Segmentation Evaluation Criteria

In this paper, three metrics are used to evaluate the segmentation result [7] [22] : DSC (Dice coefficient), HD ( Hausdorff distance) and AVD (Absolute Volume Difference), which are calculated for each tissue type(CSF, GM and WM), respectively. First, DSC is the most widely used metric in the evaluation of medical volume segmentations. In addition to the direct comparison between automatic and ground truth segmentations, it is common to use the DSC to measure reproducibility (repeatability) [23]. DSC is computed by:

(15)

Where TP, FP and FN are the subjects of true positive, false positive and false negative predictions for the considered class.

Second, the distance between crisp volumes (HD) between two finite point sets A and B is defined by:

(16)

Where is called the directed HD and given by:

(17)
(18)

Finally, the AVD is defined by:

(19)

Where A is ground truth and B is predicted volume of one class.

4.3 Experimental Results

4.3.1 Experiment one:Parameter N for Precision Analysis

In this experiment, we extract the Jacobian determinant and the curl vector of MRI brain images(T1 modality) and use them as one CNN channel with other modalities(T1-weighted, T1-IR and T2-FLAIR) respectively.We set the size(N) of grid images(JD and CV) as 64,128 and 256 and test them based on IBSR datasets for precision analysis of MRI brain segmentation.We use subject 10-14 of IBSR as training set and other subjects as testing set. We can see that the accuracy of MRI segmentation becomes better as the size of JD images(N) increases(Dice:0.6955,0.8350,0.8691 for N=64,128,256 respectively)in the following table.This also can be seen from the improvement of HD and AVD(the smaller, the better).

Experiment Dice average HD average AVD average

N=64
0.6955 11.5854 0.2755


N=128
0.8350 4.8136 0.1107

N=256
0.8691 2.7049 0.0777


Table 1: Test average results of IBSR training set(10-14) for different N of JD images with three modalities(DC:%, HD: mm, AVD:%).

Moreover,we can also see the same results based on different N of CV images with three modalities.So we prove that the result can be used in improving the accuracy of MRI segmentation by increasing the parameter N.

Experiment Dice average HD average AVD average

N=64
0.7805 5.3820 0.1846


N=128
0.8404 3.0533 0.1181

N=256
0.8569 3.4245 0.0922


Table 2: Test average results of IBSR training set(10-14) for different N of CV images with three modalities(DC:%, HD: mm, AVD:%).

4.3.2 Example two:Parameter dt for Precision Analysis

In this experiment, we use subject 4,5,7,14,070 of MRBrainS18 as training set and subject 1,148 as testing set to test the influence of the parameter dt on the MRI brain segmentation. We also compared different dt of generated JD images with three modalities.We set ntstep as 1,5,10,50,100(the corresponding dt:1,0.2,0.1,0.02,0.01) respectively.We can see that the accuracy become better as dt increases at first small steps(Dice:0.8486,0.8590,0.8669 for 1,5,10).But as dt continues to increase,the accuracy increases slowly and tends to be gentle.So we can say that the parameter dt can improve the accuracy of MRI brain segmentation within limit.

Experiment Dice average HD average AVD average

ntstep=1
0.8486 1.3441 0.0526


ntstep=5
0.8590 1.1381 0.0351

ntstep=10
0.8669 1.1220 0.0438

ntstep=50
0.8627 1.1667 0.0583

ntstep=100
0.8655 1.1381 0.0364


Table 3: Average results of MRBrainS18 training set(4,5,7,14,070) for different dt of JD images with three modalities(DC:%, HD: mm, AVD:%).

4.3.3 Example three:Parameter h for Precision Analysis

In this experiment, we use subject 4,5,7,14,070 of MRBrainS18 as training set and subject 1,148 as testing set to test the influence of the parameter h on the MRI brain segmentation. We compare different h(the spacing between points in each direction) of generated JD images with three modalities.We set h as 1,5,10,100 respectively and we can see that the accuracy of segmentation become worse with the parameter h increasing(Dice:0.8677,0.8658,0.8624,0.8469).So we can conclude that the parameter h has a influence on the accuracy of MRI segmentation.

Experiment Dice average HD average AVD average

h=1
0.8677 1.0690 0.0424

h=5
0.8658 1.1667 0.0477

h=10
0.8624 1.1381 0.0303

h=100
0.8469 1.4417 0.0756


Table 4: Average results of MRBrainS18 training set(4,5,7,14,070) for different h of JD images with three modalities(DC:%, HD: mm, AVD:%).

4.3.4 Comparison between VoxResNet and 3D-UNet on MRBrainS

In the experiment, we compare the segmentation performance of our method in two networks, VoxResNet and U-Net network. We also use subject 4,5,7,14,070 of MRBrainS18 as training set and subject 1,148 as testing set. The results show VoxResNet has a little better performance with average results(Dice:, HD:, AVD:)than U-Net network with our method in brain MRI segmentation and increase average Dice by about 1% and this result can be also seen from HD and AVD from Table 9, especially in the case of three modalities with JD, CV or both. While VoxResNet has weaker performance in the case of single or three modalities.

Experiment Dice HD AVD
VoxResNet U-Net VoxResNet U-Net VoxResNet U-Net
Single modality(T1) 0.8459 0.8669 1.3797 1.9917 0.0505 0.0397

Three modalities
0.8529 0.8548 1.3441 2.2270 0.0602 0.0479

Three modalities+JD
0.8677 0.8526 1.0690 2.1578 0.0424 0.0464

Three modalities+CV
0.8665 0.8547 1.1381 2.1075 0.0428 0.0506

Three modalities+JV
0.8648 0.8566 1.1667 2.0191 0.0483 0.0468
average 0.8596 0.8571 1.2195 2.1006 0.0488 0.0463

Table 5: Test results comparison of MRBrainS18 training set(4,5,7,14,070) for different experiments between VoxResNet and U-Net(DC:%, HD: mm, AVD:%).

We also get the same results based on MRBrainS13 by using subject 1,3,4 of MRBrainS13 as training set and subject 2,5 as testing set from Table 9.But VoxResNet has much higher performance with average results(Dice:, HD:, AVD:)than U-Net network(Dice:, HD:, AVD:) with our method in MRBrainS13.

Experiment Dice HD AVD
VoxResNet U-Net VoxResNet U-Net VoxResNet U-Net
Single modality(T1) 0.7989 0.6469 1.9267 3.9520 0.1009 0.1297

Three modalities
0.8100 0.5448 1.7475 4.8834 0.0587 0.2530

Three modalities+JD
0.8106 0.5438 1.9351 4.9356 0.0731 0.2575

Three modalities+CV
0.8249 0.5220 1.7761 5.2546 0.0193 0.2899

Three modalities+JV
0.8272 0.5601 1.8032 4.9716 0.0413 0.2626
average 0.8143 0.5635 1.8377 4.7994 0.0587 0.2385
Table 6: Test average results comparison of MRBrainS13 training set(1,3,4) for different experiments between VoxResNet and U-Net(DC:%, HD: mm, AVD:%).

5 Implementation Details

Our method was implemented using MATLAB and a flexible framework neural networks named Chainer in Python. we used MATLAB to generate the images formed by JD and CV information based on brain MRI images, and saved it as nii image format. It took about 8 hours to train the network while less than 3 minutes for processing each test volume(size ) using one NVIDIA Quadro P2000 GPU. Due to the limited GPU memory, we cropped volumetric regions(size , m is the number of image modalities and set as 2,6,8,8,10 for single modality, three modalities, three modalities+JD, three modalities+CV, three modalities+JD+CV respectively in our experiments) for the input into the network. This was implemented in an on-the-fly way during the training and the probability map of whole volume was generated in an overlap-tiling strategy for stitching the sub-volume results.

6 Conclusions

In this paper, we review the deformation method focus on the construction of diffeomorphisms, address clearly a new formation of the deformation problem for moving domains, and we apply it in natural images, face images and MRI brain images.Based on calculus of variation and optimization, we proposed a new variational method with prescribed Jacobian determinant and curl vector and use them as one CNN channel with other modalities to get more accurate results of brain segmentation. More importantly, we discuss the influence of some optimization parameters to precision analysis of MRI brain segmentation by both numerical experiments and theoretical analysis.We test this method on the IBSR dataset and MRBrainS18 dataset based on VoxResNet and prove the influence of three parameters on the accuracy of MRI brain segmentation. Finally, we compare the segmentation performance of our method in two networks, VoxResNet and 3d U-Net network. We believe the proposed method can advance the performance in brain segmentation and clinical diagnosis. In the future, we will investigate the performance of our method on more object detection and segmentation tasks.

7 Acknowledgments

The authors would like to thank Yang Deng and Yao Sun of Graduate School at Shenzhen, Tsinghua University, Mingwang Zhu of Beijing Sanbo Brain Hospital for their technical support.

References

  • [1] Z. Akkus, A. Galimzianova, A. Hoogi, D. L. Rubin, and B. J. Erickson, “Deep learning for brain mri segmentation: State of the art and future directions,” Journal of Digital Imaging, vol. 30, no. 4, pp. 449–459, 2017.
  • [2] S. Bao and A. C. S. Chung, “Multi-scale structured cnn with label consistency for brain mr image segmentation,” vol. 6, no. 1, pp. 1–5, 2015.
  • [3] A. D. Brébisson and G. Montana, “Deep neural networks for anatomical brain segmentation,” in Computer Vision and Pattern Recognition Workshops, 2015, pp. 20–28.
  • [4] H. Chen, Q. Dou, L. Yu, J. Qin, and P. A. Heng, “VoxResNet: Deep Voxelwise Residual Networks for Volumetric Brain Segmentation,” Neuroimage, vol. 170, pp. 446–455, 2017.
  • [5] X. Chen and G. Liao, “New variational method of grid generation with prescribed jacobian determinant and prescribed curl,” Computer Science, pp. 2–6, 2015.
  • [6] J. Dai, K. He, and J. Sun, “Instance-aware semantic segmentation via multi-task network cascades,” in Computer Vision and Pattern Recognition, 2016, pp. 3150–3158.
  • [7] Y. Deng, Y. Sun, Y. Zhu, M. Zhu, W. Han, and K. Yuan, “A strategy of mr brain tissue images’ suggestive annotation based on modified u-net,” pp. 5–7, 2018.
  • [8] Q. Dou, H. Chen, L. Yu, L. Zhao, J. Qin, D. Wang, V. Mok, L. Shi, and P. A. Heng, “Automatic detection of cerebral microbleeds from mr images via 3d convolutional neural networks,” IEEE Transactions on Medical Imaging, vol. 35, no. 5, pp. 1182–1195, 2016.
  • [9] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” pp. 770–778, 2015.
  • [10] K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual networks,” in European Conference on Computer Vision, 2016, pp. 630–645.
  • [11] G. Liao, F. Liu, G. C. De, la Pena, D. Peng, and S. Osher, “Level-set-based deformation methods for adaptive grids,” Journal of Computational Physics, vol. 159, no. 1, pp. 103–122, 2000.
  • [12] D. lin, A. V. Vasilakos, Y. Tang, and Y. Yao, “Neural networks for computer-aided diagnosis in medicine,” Neurocomputing, vol. 216, pp. 700–708, 2016.
  • [13] F. Liu, S. Ji, and G. Liao, An Adaptive Grid Method and Its Application to Steady Euler Flow Calculations.   Society for Industrial and Applied Mathematics, 1998, vol. 20, no. 3.
  • [14] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3431–3440.
  • [15] O. Maier, B. H. Menze, J. V. D. Gablentz, L. Häni, M. P. Heinrich, M. Liebrand, S. Winzeck, A. Basit, P. Bentley, and L. Chen, “Isles 2015 - a public evaluation benchmark for ischemic stroke lesion segmentation from multispectral mri,” Medical Image Analysis, vol. 35, pp. 250–269, 2017.
  • [16] A. M. Mendrik, K. L. Vincken, H. J. Kuijf, M. Breeuwer, W. H. Bouvy, B. J. De, A. Alansary, B. M. De, A. Carass, and A. El-Baz, “Mrbrains challenge: Online evaluation framework for brain image segmentation in 3t mri scans,” Computational Intelligence and Neuroscience, vol. 2015, no. 4-5, p. 813696, 2015.
  • [17] B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, Y. Burren, N. Porz, J. Slotboom, and R. Wiest, “The multimodal brain tumor image segmentation benchmark (brats),” IEEE Transactions on Medical Imaging, vol. 34, no. 10, pp. 1993–2024, 2015.
  • [18] P. Moeskops, M. A. Viergever, A. M. Mendrik, L. S. de Vries, M. J. Benders, and I. Isgum, “Automatic segmentation of mr brain images with a convolutional neural network.” IEEE Transactions on Medical Imaging, vol. 35, no. 5, pp. 1252–1261, 2016.
  • [19] D. Nie, L. Wang, Y. Gao, and D. Shen, “Fully convolutional networks for multi-modality isointense infant brain image segmentation,” Proc IEEE Int Symp Biomed Imaging, vol. 108, pp. 1342–1345, 2015.
  • [20] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” Medical Image Computing and Computer-Assisted Intervention, vol. 9351, pp. 234–241, 2015.
  • [21] M. F. Stollenga, W. Byeon, M. Liwicki, and J. Schmidhuber, “Parallel multi-dimensional lstm, with application to fast biomedical volumetric image segmentation,” Computer Science, p. 2998–3006, 2015.
  • [22]

    Y. Sun, Y. Deng, Y. Xu, S. Zhang, M. Zhu, and K. Yuan, “A multi-channel network with image retrieval for accurate brain tissue segmentation,” pp. 2–5, 2018.

  • [23] A. A. Taha and A. Hanbury, “Metrics for evaluating 3d medical image segmentation: analysis, selection, and tool.” Bmc Medical Imaging, vol. 15, no. 1, p. 29, 2015.
  • [24] W. Zhang, R. Li, H. Deng, L. Wang, W. Lin, S. Ji, and D. Shen, “Deep convolutional neural networks for multi-modality isointense infant brain image segmentation,” NeuroImage, vol. 108, pp. 214–224, 2015.