Morphological feature visualization of Alzheimer's disease via Multidirectional Perception GAN

11/25/2021
by   Wen Yu, et al.
0

The diagnosis of early stages of Alzheimer's disease (AD) is essential for timely treatment to slow further deterioration. Visualizing the morphological features for the early stages of AD is of great clinical value. In this work, a novel Multidirectional Perception Generative Adversarial Network (MP-GAN) is proposed to visualize the morphological features indicating the severity of AD for patients of different stages. Specifically, by introducing a novel multidirectional mapping mechanism into the model, the proposed MP-GAN can capture the salient global features efficiently. Thus, by utilizing the class-discriminative map from the generator, the proposed model can clearly delineate the subtle lesions via MR image transformations between the source domain and the pre-defined target domain. Besides, by integrating the adversarial loss, classification loss, cycle consistency loss and L1 penalty, a single generator in MP-GAN can learn the class-discriminative maps for multiple-classes. Extensive experimental results on Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset demonstrate that MP-GAN achieves superior performance compared with the existing methods. The lesions visualized by MP-GAN are also consistent with what clinicians observe.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 4

page 7

page 8

page 10

page 11

08/03/2020

Tensorizing GAN with High-Order Pooling for Alzheimer's Disease Assessment

It is of great significance to apply deep learning for the early diagnos...
08/27/2019

Adversarial regression training for visualizing the progression of chronic obstructive pulmonary disease with chest x-rays

Knowledge of what spatial elements of medical images deep learning metho...
06/14/2019

GAN-based Multiple Adjacent Brain MRI Slice Reconstruction for Unsupervised Alzheimer's Disease Diagnosis

Leveraging large-scale healthy datasets, unsupervised learning can disco...
10/12/2021

DecGAN: Decoupling Generative Adversarial Network detecting abnormal neural circuits for Alzheimer's disease

One of the main reasons for Alzheimer's disease (AD) is the disorder of ...
11/24/2017

CatGAN: Coupled Adversarial Transfer for Domain Generation

This paper introduces a Coupled adversarial transfer GAN (CatGAN), an ef...
07/04/2020

Interpretation of Disease Evidence for Medical Images Using Adversarial Deformation Fields

The high complexity of deep learning models is associated with the diffi...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Alzheimer’s Disease (AD) is an irreversible and chronic neurodegenerative disease with progressive impairment of memory and other mental functions. It is estimated to be the fifth leading cause of death in elderly people

[16]. AD is caused by abnormal cell death in the brain, long before amnestic symptoms are observable[7]

. The resulting brain atrophy is visible in structural magnetic resonance (MR) images. To date, AD is incurable but preventable. Since machine learning tools have obtained great success in medical image computing

[56, 59], it is practicable to diagnose the early stages of AD by MR images for timely treatment[44, 57, 55] and many works [32, 22, 20, 21, 64, 31, 19] has been done to for the early stages of AD. Significant memory concern (SMC) and mild cognitive impairment (MCI) are the transitional stages between normal controls (NC) and AD[36]

. SMC and MCI present mild symptoms, and the disease-related regions are very subtle in MR images. Currently, the clinical diagnosis procedure is time-consuming and requires extensive clinical training and experience for clinicians. Thus, developing automatic methods by utilizing deep learning to visualize the brain changes for the early stages of AD is highly desirable. It can assist clinicians for AD analysis and may provide meaningful information on the pathogenesis of cognitive decline. However, this is a challenging task due to several reasons, such as the low-intensity contrast between the lesion and other neighboring regions, the indistinct boundary of the lesion, and the irregular lesion shape.

To visualize features of different Alzheimer’s stages in MR images, there already exist several feature visualization methods based on classification. These methods can be classified into two categories. (1) The Regions Of Interest (ROI)-based classification approaches

[16, 52, 27, 4] and patch-based classification approaches[34]

. The performance of these methods is limited since the brain ROIs or patches need to be selected based on anatomical brain atlases or biological prior knowledge beforehand. Multiple steps are required to exact features from ROIs or patches for classification and subsequent visualization. Therefore, they tend to be sensitive to parameters and time-consuming; (2) Three strategies to visualize features for a convolutional neural network (CNN) classifier. (i)By editing an input image and observing its effect on the prediction results, the occluded regions which have a significant impact on prediction can be visualized; (ii)By analyzing the gradients of the prediction for an input image, a heatmap can be produced for visualization; (iii)By analyzing the activations of the feature maps for the image, the regions which are responsible for making the specific prediction can be visualized. These classification-based feature visualization methods make their predictions based on local regions most relevant to the particular prediction rather than the whole image, and it may ignore features with low discriminative power if stronger features for the prediction are available. As a result, if there is evidence for a category at multiple locations in the image (such as multiple AD lesions in MR images), some lesions with low discriminative power may be ignored. Moreover, visual features strongly depend on the performance of the classifier, and a large number of labeled samples are required to train a robust model.

Fig. 1: The pipeline of the MP-GAN model. Assuming the MP-GAN is well trained, given the real EMCI patient’s MR images and the NC label, the subtle morphological features between NC and EMCI can be visualized by the model to assist clinicians for AD analysis and treatment.

To alleviate these issues, a novel Multidirectional Perception Generative Adversarial Network (MP-GAN) is proposed to visualize morphological features for whole-brain MR images as shown in Fig. 1. Generative Adversarial Network (GAN)[13, 62],deriving from variational methods [38, 54, 53]

, has attracted lots of attention as it is capable of generating realistic data without explicitly modeling the probability density function. Specifically, the generator of MP-GAN takes inputs as both MR images and its target domain. Then it flexibly learns a class-discriminative map for the target domain. By adding the class-discriminative map and the input MR image of the source domain, a synthetic MR image of the target domain can be produced. Thus the learned class-discriminative map can capture all brain changes by transforming the MR image between the source domain and the target domain. By visualizing the class-discriminative maps, the subtle and complex lesions that may not be found within one region can be identified. Besides, by designing the hybrid loss, a single generator in MP-GAN can learn the class-discriminative maps for multiple-classes. In this manner, the common features unrelated to the specific domain can be reused during training, therefore the visualization performance is further improved. With this global lesion visualization, clinicians can better exclude undesirable biases and potentially even identify previously unknown characteristics of AD. To the best of our knowledge, the proposed MP-GAN is the first work to visualize the morphological features for different Alzheimer’s stages by a single generator. The contributions of this paper are summarized as follows:

  1. A novel MP-GAN with a multidirectional mapping mechanism is proposed to capture the salient global features efficiently. By utilizing the class-discriminative map from the generator, the proposed model can clearly delineate the subtle lesions via MR image transformations between the source domain and the target domain.

  2. By integrating the adversarial loss, classification loss, cycle consistency loss and L1 penalty, a single generator in MP-GAN can learn the class-discriminative maps for multiple-classes. The morphological features indicating different Alzheimer’s stages can be visualized by a single MP-GAN model.

The rest of this paper is organized as follows. The related work is reviewed in Section II. The proposed MP-GAN is described in detail in Section III. In Section IV, MP-GAN is tested and compared with existing feature visualization methods to demonstrate its advantage. Finally, concluding remarks and future work are discussed in Section V and Section VI.

Ii Related work

Generative Adversarial Networks.

GAN has attracted lots of attention as it is capable of generating realistic data without explicitly modeling the probability density function. It has shown remarkable results in various computer vision tasks such as image generation

[13]

, image-to-image translation

[69]

, image super-resolution

[62]

, and semi-supervised learning

[65] [58]. A typical GAN model consists of two modules: a discriminator and a generator. The discriminator learns to distinguish between real and fake samples, while the generator learns to generate fake samples that are indistinguishable from real samples. Training the original GAN, however, suffers from several problems such as low quality of generated images, convergence problems, and mode collapse. To address these deficiencies, variants of the GAN were introduced[2, 15]. The most representative work is Wasserstein GAN (WGAN) [2]

. It leverages the Wasserstein distance to measure the distance between two data distribution that has better theoretical properties than the original KL (Kullback–Leibler) divergence.

Feature visualization methods.The current feature visualization methods for AD generally fall into two categories: (1)The ROI-based classification approaches; (2)The CNN-based classification approaches.

For the first category, the brain ROIs or patches were selected based on anatomical brain atlases or biological prior knowledge beforehand, then multiple steps were required to extract features from ROIs or patches for classification. According to classification performance, the most frequently selected ROIs or patches would be visualized[52, 27]. For instance, Lian et al. [34] proposed a hierarchical fully convolutional network (H-FCN) to automatically identify discriminative local patches and regions in MR images for AD analysis. The hierarchical discriminative locations of brain atrophy at both the patch-level and region-level were visualized.

For the second category, there were three strategies to visualize features for CNN.

  1. By editing an input image and observing its effect on the prediction results, the occluded regions which had a significant impact on prediction can be visualized[40]. For instance, Zeiler and Fergus [66] proposed an occlusion-based method to visualize the activity within CNN. Different portions of the input image were occluded with a grey square, and the output of the classifier was observed. The occluded regions which cause the probability of the correct class drop significantly would be visualized. Korolev et al. [30] utilized 3D-ResNet for AD classification, and the important regions of the MR image most affected by AD were visualized by the occlusion-based method [66].

  2. By analyzing the gradients of the prediction for an input image, the heatmap can be produced for visualization [35, 45, 63, 43, 1, 6]. For example, Springenberg et al. [48]

    proposed a new variant of the “deconvolution approach” guided backpropagation for visualizing features learned by CNNs. Guided backpropagation can be applied to a broader range of network structures. Sundararajan et al.

    [49] proposed Integrated Gradients by utilizing an axiomatic framework for feature visualization;

  3. By analyzing the activations of the feature maps for the image, the regions which were responsible for making the specific prediction can be visualized. For instance, Zhou et al.[68] proposed Class Activation Mapping (CAM) to visualize the discriminative object parts detected by CNN in a single forward pass. Khan et al. [28]

    utilized VGG with transfer learning for AD analysis. CAM was utilized to visualize the discriminative regions in the MR image for model interpretation. Lian et al.

    [33] proposed a multi-task weakly-supervised attention network (MWAN) by leveraging a fully-trainable dementia attention block for regression. The attention maps were visualized by CAM for AD subjects. Sarraf et al.[42] utilized LeNet-5 to classify structural MR images for AD vs. NC. The filters and the features were visualized for interpretation.

Iii The Proposed MP-GAN

Iii-a Overview

This paper proposes a novel mapping mechanism by which the MR images can be mapped between each pair of source class and target class in a multidirectional manner. Take the source class NC as an example, the generator can map the MR images between NC and SMC, meanwhile, it can also map MR images between NC and the other class such as EMCI, LMCI, and AD simultaneously. The flowchart of MP-GAN is shown in Fig. 2. After data preprocessing (see Section IV-A

), the normalized T1-MR images of all classes are fed into MP-GAN. The proposed model learns the class-discriminative maps between all class-pairs for visualizing morphological features. More specifically, the generator aims to capture salient global features in class-discriminative maps. Then the class-discriminative maps are used to transform MR images between the source domain and the target domain. To control semantic information, an auxiliary classifier is introduced based on the generator and discriminator to form the MP-GAN architecture. While the generator produces the class-discriminative maps distinguishing between the source domain and the target domain, the classifier predicts the domain indicating Alzheimer’s stage, and the discriminator identifies whether the transformed MR images are real or fake. In this manner, the class-discriminative maps learned by MP-GAN can highlight exactly which regions of the MR image are significant for discrimination between the source domain and the target domain at the voxel-level. The subtle and complex lesions that may not be found within one region can be identified. Furthermore, since the input MR images are high-order with complicated brain structure, MP-GAN is further designed with the following two improvements: (1) 3D Residual Blocks are exploited in the conditional generator so that the features from the low-level can be reused, and the vanishing-gradient problem can be prevented; (2) 3D-DenseNet is utilized in classifier to capture more discriminative features.

Fig. 2: The flowchart of MP-GAN. It consists of three components: a generator, a classifier, and a discriminator. The generator maps the input source MR image to the synthetic target MR images by the class-discriminative map. The synthetic source MR images are reconstructed from the synthetic target MR images by the generator in the same manner. By utilizing classification loss, adversarial loss, and cycle consistency loss, the generator learns to generate synthetic target MR images and the reconstructed source MR images that are indistinguishable from real MR images.

Iii-B The Architecture

The proposed MP-GAN is designed to visualize morphological features for multiple-classes. To achieve this, the generator G is designed to produce class-discriminative map which can transform an input MR image to an output MR image conditioned on the target class , . During training, the target class is randomly selected so that G learns to produce class-discriminative maps for all class-pairs. By doing so, the target class can be predefined, and global features that distinguish between the source domain and the desired target domain can be visualized at the testing stage.

As illustrated in Fig. 2, input MR image is labeled and represents the corresponding class. The conditional generator aims to capture all salient global features in class-discriminative maps . Then is utilized to transform input MR image from the source domain to the target domain in a bidirectional manner. The classifier predicts label given real MR image by the conditional distribution , and the discriminator is trained to identify whether the MR image is real or fake. Formally, given an MR image of source class and a conditional variable , the generator can produce a synthetic MR image of target class by adding the generated class-discriminative map and input MR image .

(1)

which is indistinguishable from the real MR image of the target domain . Thereby, class-discriminative map contains all salient global features which distinguish between two domains and . The change of salient voxels between the source domain and the target domain on the MR image can be visualized by the class-discriminative map.

Adversarial Loss. To make the synthetic target MR images indistinguishable from real MR images, an adversarial loss is defined as

(2)

where generator G generates an MR image conditioned on both the input MR image and the target class , while discriminator D attempts to distinguish between real and fake MR images. The G tries to minimize this adversarial loss, while the D tries to maximize it. More specifically, when the discriminator successfully identifies real and fake MR images, it is rewarded and no change is needed to update the parameters of the discriminator, whereas the generator is penalized with large updates to parameters. Alternately, when the generator fools the discriminator, it is rewarded, and no change is needed to update the parameters of the generator, but the discriminator is penalized and its component parameters are updated.

Classification Loss. Given an input MR image and a target class , the goal of MP-GAN is to produce a class-discriminative map that can transform into an output MR image . aims to be classified as the target class

. To achieve this condition, an independent classifier is introduced and the classification loss is imposed when optimizing generator G. Specifically, the loss function is decomposed into two terms: a classification loss of real images to optimize classifier C, and a classification loss of fake images to optimize generator G. In detail, the former is defined as

(3)

By minimizing this classification loss, classifier C learns to classify a real MR image to its corresponding class . On the other hand, the loss function for the classification of fake images is defined as

(4)

Generator G tries to minimize the loss to produce the class-discriminative maps for generating MR images that can be classified as the target class .

Cycle consistency loss. By minimizing the adversarial and classification losses, generator G is trained to generate MR images that are realistic and classified as target class. However, minimizing the losses (Eqs. (2) and Eqs. (4)) does not guarantee that the final transformed images preserve the content of input MR images while changing only the disease-related regions of the input. To alleviate this problem, a forward cycle consistency loss and backward cycle consistency loss [29, 69] are applied to the generator. They are defined as

(5)
(6)

where generator G takes in the transformed MR image and the source class as input and tries to reconstruct the MR image of the source domain . The L1 norm is adopted as the reconstruction loss. Note that a single generator is reused twice. The generator is first utilized to transform MR images from the source domain to MR images of the target domain . Then it is used to reconstruct the MR image of the source domain from the synthetic MR images of the target domain . For the first utilization, forward cycle consistency loss is adopted. For the second one, backward cycle consistency loss is adopted.

L1 penalty. The smallest class-discriminative map that leads to a real MR image of the target domain is encouraged. Thus L1 penalty is defined as

(7)

where is the norm.

Total Loss. The total loss functions to optimize D, C, and G are defined respectively as

(8)
(9)
(10)

where , , and

are hyperparameters that control the relative importance of classification loss,

L1 penalty, and cycle consistency loss respectively, compared to the adversarial loss. is set as 0.1, is set as 10, is set as 10 and is set as 1 empirically throughout the paper. Please note that each loss term is indispensable for the proposed MP-GAN. Without any loss term of the hybrid loss, the training of MP-GAN will become extremely unstable and the learned class-discriminative maps can’t capture the salient features for each pair of classes.

Fig. 3: Network architecture of the proposed MP-GAN. It consists of 3 components:(a) Generator, (b) Classifier, and (c) Discriminator. The generator consists of two convolutional layers for downsampling, three 3D-residual blocks, and two transposed convolutional layers for upsampling. 3D-DenseNet is utilized in the classifier. A standard CNN architecture with 7 convolutional layers with and convolutional filters is adopted in the discriminator.

During the training process, the following domain settings are defined to train so that all features between any two domains and can be visualized for AD analysis.

(1) ={NC}, ={SMC,EMCI,LMCI,AD},

(2) ={SMC}, ={NC,EMCI,LMCI,AD},

(3) ={EMCI}, ={SMC,NC,LMCI,AD},

(4) ={LMCI}, ={SMC,EMCI,NC,AD},

(5) ={AD}, ={SMC,EMCI,LMCI,NC}.

From the algorithm perspective, take the first case (1) as an example, when source domain is set as NC, the target domain will be set as one of {SMC, EMCI, LMCI, AD}. Please note that all categories in {SMC, EMCI, LMCI, AD} set will be trained at least once. In this way, all the above 5 conditions will be trained for MP-GAN, thus a single generator in MP-GAN can learn the class-discriminative maps for each pair of classes, and the salient global features can be captured for multiple classes. At the testing stage, is pre-defined according to the requirement of user. In this paper, at the testing stage, the morphological features of NC versus all Alzheimer’s stages including SMC, EMCI, and LMCI are visualized. MCI is characterized by a slight decline in cognitive abilities. Note that patients with MCI are at increased risk of developing to AD, but do not always do. Thus MCI is significant for morphological feature visualization and further AD analysis.

The network structure of generator, classifier, and discriminator is shown respectively in Fig. 3. The network utilized in the generator is ResNet. 3D-ResNet is expanded by adding a spatial dimension to all convolutional and pooling layers in ResNet for the MR image. By utilizing the shortcut connection, ResNet explicitly reformulates the layers as learning residual functions regarding the input layer, and it transfers feature representations from low layers to the high layers. More specifically, assume the target class

is a discrete label, and it is encoded as a one-hot tensor. The target label

is concatenated to the input MRI tensor in a depth-wise manner. Then they are operated by two convolutional layers with a stride size of 2 for downsampling, three 3D-residual blocks

[17], and two transposed convolutional layers with the stride size of 2 for upsampling. In this manner, the target label is operated with the input MRI tensor of the source label in each hidden layer of the generator, and the class-discriminative map between y and will be generated. Finally, a synthetic MR image of target class is produced by adding the generated class-discriminative map and input MRI tensor x. Instance normalization[51] is used in all layers except the last output layer for the generator. and convolutional filters are employed in generator. The network utilized in the classifier is DenseNet [23]. 3D-DenseNet is expanded by adding a spatial dimension to all convolutional and pooling layers in DenseNet for the MR image. Feature-maps learned by all preceding layers are concatenating along the last dimension for the subsequent layers. Through such dense connectivity, feature-maps are reused and the vanishing-gradient problem is alleviated. Meanwhile, 3D-DenseNet can extract discriminative features related to Alzheimer’s stage from the whole MR images efficiently. The details of 3D-denseNet can be found in [23, 14]. In this paper, the depth is set to 30, the growth rate is set to 12, the number of the Dense-BC block is set to 3, and the reduction is set to 0.5. A standard CNN architecture with 7 convolutional layers with and

convolutional filters is adopted in the discriminator. Each convolutional layer is followed by batch normalization

[24]

and ReLU.

Magnet strength 3T 1.5T
Source ADNI-1 ADNI-GO ADNI-2 ADNI-1 ADNI-GO ADNI-2
Subject NC AD EMCI NC SMC EMCI LMCI AD NC AD NC NC
Number 42 29 142 190 121 309 177 159 171 175 16 81
Gender(F/M) 27 / 15 20 / 9 67 / 75 95 / 95 64 / 47 139 / 169 80 / 97 68 / 91 89 / 82 85 / 90 8 / 8 45 / 36
Age 76.15.1 75.67.9 71.77.7 74.96.8 72.95.6 72.37.3 72.97.6 75.47.9 77.75.4 76.67.5 80.24.8 82.54.5
Education 162.8 14.72.9 15.82.7 16.42.7 16.82.5 162.7 16.52.6 15.82.7 16.02.9 14.63.2 15.52.5 15.92.9
MMSE 29.31.0 20.034.8 28.21.8 28.71.5 28.61.7 28.02.1 263.5 20.84.4 29.11.2 21.54.4 29.41.0 28.52.6
CDR 00.14 1.070.4 0.450.19 0.070.19 0.130.23 0.460.22 0.580.37 0.990.46 00.19 0.930.49 0.070.18 0.20.35
Samples 149 73 471 723 288 1111 616 501 587 520 72 205
TABLE I: Demographic characteristics of the subjects in ADNI dataset.

Iv Experiments and Results

Iv-a Dataset and Preprocessing

There are five stages associated with AD progression: Normal Control (NC), Significant memory concern (SMC), early mild cognitive impairment (EMCI), late mild cognitive impairment (LMCI), and Alzheimer’s disease(AD). T1-weighted MR images from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) public dataset are used for the evaluation purpose. 5316 MR images in ADNI-1, ADNI-go, and ADNI-2 are utilized. It includes 1736 NC subjects, 288 SMC subjects, 1582 EMCI subjects, 616 LMCI subjects, and 1094 AD subjects. Both 1.5T and 3T field strength MR images are used. Table I lists the demographic characteristics of the subjects.

ADNI-go and ADNI-2 added 129 and 782 participants respectively to the 819 recruited by ADNI-1111 http://adni.loni.usc.edu/about/. In ADNI-1, NC and MCI participants continue to be followed by ADNI-go and ADNI-2. Different from the ADNI-1 dataset, MCI is divided into two subtypes, including EMCI, and LMCI in the ADNI-2 dataset. ADNI-go also added a new cohort of people with EMCI, and ADNI-2 added a cohort who were clinically evaluated as cognitively normal but had SMC. Note that SMC is the transitional stage between NC and MCI. The diagnostic criteria are described in the ADNI procedures manual222http://www.adni-info.org.

All MR images are in the neuroimaging informatics technology initiative (NIfTI) format. They are processed using standard operations in the FSL333www.fmrib.ox.ac.uk/fsl toolbox [61, 46, 47] for registering the MR images to MNI space. The preprocessing pipelines contain three parts: (1) removal of redundant tissues; (2) brain area extraction by BET; (3) linear registration by FLIRT [26, 25]. Lastly, the T1-MR image is normalized into the range [-1,1], and is fed into the MP-GAN model as a tensor directly without compressing or downsizing.

Iv-B Experiment Settings

The proposed MP-GAN is trained on the ADNI dataset from scratch in an end-to-end manner. All methods are implemented in TensorFlow

444http://www.tensorflow.org/

. All experiments are conducted on four NVIDIA GeForce GTX 2080 Ti GPUs. ‘Adam’ is utilized as the optimizer for stochastic gradient descent (SGD). The batch size is set to 8 empirically as each MR image is a high-order tensor of

. Since the batch size is relatively small, then the gradients will become unstable, thus there is a need to reduce the learning rate to stabilize the training. According to the experimental results, the learning rate of both generator and classifier is set to 0.001, and the learning rate of the discriminator is set much smaller as

. For evaluation, 80% of the MR images are allocated for training. The remaining 20% of the MR images are equally partitioned and used as validation and test data sets respectively. For avoiding bias, the training set, validation set, and test set do not have the MR images from the same subject simultaneously. A single MP-GAN model is trained on a training dataset of all categories, then the morphological features between the source domain and the predefined target domain are visualized on the test set. The validation dataset is utilized to tune hyperparameters to obtain the best model out of several epochs during the training process.

Fig. 4: Heatmaps predicted by Integrated Gradients, Guided Backprop, CAM, GAN, and our method are shown in sagittal,coronal, and axial views for NC vs. SMC respectively.
Fig. 5: Heatmaps predicted by Integrated Gradients, Guided Backprop, CAM, GAN, and our method are shown in sagittal,coronal, and axial views for NC vs. EMCI respectively.
Fig. 6: Heatmaps predicted by Integrated Gradients, Guided Backprop, CAM, GAN, and our method are shown in sagittal,coronal, and axial views for NC vs. LMCI respectively.
Fig. 7: Heatmaps predicted by Integrated Gradients, Guided Backprop, CAM, GAN, and our method are shown in sagittal,coronal, and axial views for NC vs. AD respectively.

Iv-C Qualitative Analysis

In this section, comprehensive experiments are conducted to show the effectiveness of MP-GAN. First, the proposed model is compared with 4 methods: (1)Guided Backpropagation[48]; (2)Integrated Gradients[49]; (3)Class Activation Mapping(CAM)[68]; and (4)GAN [5]

. For Guided Backpropagation, Integrated Gradients, and CAM, a conventional CNN architecture is used for these networks. More specifically, the CNN architecture consists of 10 convolutional layers followed by the batch normalization and max-pooling layers. After the last convolutional layer, an average pooling layer is utilized instead of the fully connected layer. Besides, for the CAM method, the last layer is designed as described in

[68] and the last two max-pooling layers are omitted. This allows for more accurate heatmaps due to the higher resolution of the last feature maps. The proposed method is also compared with the conventional GAN [5] to demonstrate our advantages. For a fair comparison, the network structure of the generator and discriminator in GAN is the same as the proposed MP-GAN, and the loss function of GAN is the conventional adversarial loss. The GAN is trained to visualize important regions in MR images between two predefined classes. Furthermore, the following 4 evaluation groups are set up when compared with the 4 existing methods: (1)NC vs. SMC, (2)NC vs. EMCI, (3)NC vs. LMCI, and (4)NC vs. AD. The main reason for this setup is that more meaningful pathological features can be found by comparing with healthy people. It is worth noting that MR images of all 5 classes are trained using only one MP-GAN model, and the class-discriminative map for each evaluation group is visualized at the test stage. But for the 4 compared methods, one independent binary model is trained for each evaluation group respectively.

To visually show the quality of heatmaps produced by the proposed model and the 4 existing methods, one MR image is taken from each evaluation group for qualitative analysis. From Fig. 4 to Fig. 7, the heatmaps from the sagittal, coronal, and axial views are illustrated for each evaluation group respectively. The figures are shown by progression from SMC to AD in order. From Fig. 4 to Fig. 7, it can be seen that the proposed MP-GAN can visualize subtle lesions with contour edge at a finer scale (i.e., voxel-level). More detailed discriminative regions can be depicted, such as the hippocampus, and the corners and boundaries of the ventricle. The highlighted subtle lesions predicted by MP-GAN are relatively more precise than those generated by the other 4 methods. For example, from Fig. 5, it can be observed that the lesions that have much more blurred boundary and are difficult to recognize can be delineated by MP-GAN. More specifically, the corpus callosum with irregular sulcus is depicted accurately by MP-GAN from the sagittal-view and coronal-view in Fig. 5. Atrophy of the corpus callosum may lead to functional disability because of reduced interhemispheric integration. It is a region that has been examined intensively for indications of EMCI [3]. On the other hand, Integrated Gradients and Guided Backprop tend to focus on some small parts of the lesions rather than the whole lesions. Because some subtle voxels of the lesion might be more salient than the other voxels of the whole lesion. This proves that the feature visualization methods based on classification only focus on the most discriminative features and ignore the rest. It is difficult to interpret the results produced by CAM due to the low-resolution. Moreover, the regions visualized by GAN seem to cover parts of ground-truth affected by the AD for NC vs. AD as shown in Fig. 7. However, they are not close to ground-truth, this is because the training of GAN is unstable. In summary, the results of the proposed MP-GAN are closer to the ground-truth compared with the other 4 existing methods. This implies that MP-GAN can benefit from the multidirectional mapping mechanism and the hybrid loss function. MP-GAN is more sensitive to subtle structural changes in MR images caused by cognitive decline.

The ADNI diagnostic criteria for each Alzheimer’s stage are briefly described as following. (1) NC participants have no subjective or informant-based complaints of memory decline, and they have a normal cognitive performance; (2) SMC participants have subjective memory concerns assessed by the Cognitive Change Index(CCI). They have no informant-based complaint of memory impairment or decline, and they have a normal cognitive performance on the Wechsler Logical Memory Delayed Recall (LM-delayed) and the Mini-Mental State Examination (MMSE)[39]

; (3) EMCI participants have a subtle cognitive decline. Their abnormal memory function is approximately 1 standard deviation below normative performance, and their MMSE total score is greater than 24; (4) LMCI participants have a memory concern. Clinical Dementia Rating (CDR) of LMCI participants is 0.5, and the Memory Box (MB) score must be at least 0.5; (5) AD participants have a significant memory concern. The MMSE score of AD participants is between 20 and 26, and CDR is 0.5 or 1.0.

To further analyze the visualization results of the proposed MP-GAN from a clinical perspective, the two-view slices in another coordinate of (33,55,39) are shown in Fig. 8. Note that the three-view slices shown from Fig. 4 to Fig. 7 are in the coordinate of (44,55,47). From Fig. 8, the following observations can be made. (1) For all four evaluation groups, the proposed MP-GAN can delineate the discriminative lesions clearly. More specifically, lesions visualized by MP-GAN are hippocampus, thalamus, putamen, pallidum, caudate nucleus, amygdala, and insula[67, 41, 30]. It is worth noting that the discriminative capability of these brain regions in clinical analysis has already been validated by previous studies[11, 16, 8, 10]. This implies the feasibility of the proposed MP-GAN; (2) The morphological changes including global atrophy (e. g. smaller volumes of hippocampus or amygdala) and shape changes are visualized by class-discriminative map (indicated by color). These morphological changes are related to AD disease progression and cognitive decline severity; (3) For 4 evaluation groups, identified multiple regions are overlapped or localized at similar brain regions. For instance, the regions of NC vs. LMCI and NC vs. AD are similar because LMCI might develop to AD. Meanwhile, since the features between LMCI and AD are very subtle, thus some visualized regions of NC vs. LMCI and NC vs. AD are overlapped, but the atrophy severity of each lesion is different (indicated by color). The lesions visualized for EMCI vs. AD and NC vs. AD also have some common regions, such as the hippocampus and pallidum. Furthermore, it is reasonable that the overlap regions between NC vs. EMCI and NC vs. AD might not be identified for EMCI vs. AD, and some regions such as the amygdala which are specific to EMCI vs. AD can be identified; (4) Along with the progression from EMCI to AD, from Fig. 8(a) to Fig. 8(c), it can be observed that the intensity values (i.e., light salmon color) in the heatmaps are gradually increased (i.e., change to crimson) at various brain locations, and some of them are accumulated at the annotated regions. These results suggest that the class-discriminative maps generated by the proposed MP-GAN have the potential to provide some extra information regarding the AD progression, and it may reveal the gradual atrophic process of the human brain due to cognitive decline. Furthermore, the severity of cognitive decline is also reflected in ADNI diagnostic criteria for each Alzheimer’s stage as described above. In summary, the above observations imply the robustness of MP-GAN in visualizing morphological features for different Alzheimer’s stages.

Fig. 8: Class-discriminative maps generated by MP-GAN are shown as a colored overlay over the MR images. The regions affected by the progression of AD are reliably captured by MP-GAN for four evaluation groups respectively.
Fig. 9: Distribution of class-discriminative maps visualized by MP-GAN for five evaluation groups respectively.

For further visualization analysis, 5 evaluation groups are investigated respectively in Fig. 9. The results show that the important brain regions visualized by the proposed method are consistent with regions in Fig. 8. More specifically, by aligning the automatic anatomical labeling (AAL) map with the class-discriminative maps visualized in Fig. 9, each region in the class-discriminative map will be matched to the specific ROI index and name in AAL. The disease-related regions visualized by MP-GAN are listed in Table II. Note that the suffix ‘L’ denotes the left brain, and the suffix ‘R’ denotes the right brain. The following observations can be made from Fig. 9 and Table II. (1) The brain regions visualized by the proposed method for NC vs. SMC are precental gyrus, middle frontal gyrus, inferior frontal gyrus, median cingulate, paracingulate gyri, parahippocampal gyrus, superior occipital gyrus, postcentral gyrus and thalamus; (2) The brain regions visualized by the proposed method for SMC vs. EMCI are rolandic operculum, insula, parahippocampal gyrus, amygdala, superior occipital gyrus, middle occipital gyrus, postcentral gyrus, superior parietal gyrus and precuneus; (3) The brain regions visualized by the proposed method for NC vs. EMCI are the middle frontal gyrus, posterior cingulate gyrus, calcarine fissure and surrounding cortex, cuneus, superior occipital gyrus, fusiform gyrus, postcentral gyrus, lenticular nucleus, putamen and inferior temporal gyrus; (4) The brain regions visualized by the proposed method for EMCI vs. LMCI are the superior frontal gyrus, orbital part, inferior frontal gyrus, opercular part, hippocampus, parahippocampal gyrus, calcarine fissure and surrounding cortex, lingual gyrus, inferior occipital gyrus and fusiform gyrus; (5) The brain regions visualized by the proposed method for LMCI vs. AD are the middle frontal gyrus, orbital part, inferior frontal gyrus, triangular part, hippocampus, calcarine fissure and surrounding cortex, lingual gyrus, middle occipital gyrus, precuneus, lenticular nucleus and putamen. These regions also agree with the existing research findings. To sum up, the lesions visualized by the proposed model are highly suggestive and effective for tracking the progression of AD.

Fig. 10: Visualization results of hippocampus by MP-GAN in sagittal view and corresponding zoomed regions. The subfigures at the bottom are the zoom of the original subfigures for better observation.
NC vs. SMC SMC vs. EMCI NC vs. EMCI EMCI vs. LMCI LMCI vs. AD
ROI index ROI name ROI index ROI name ROI index ROI name ROI index ROI name ROI index ROI name
1 Precentral_L 17 Rolandic_Oper_L 7 Frontal_Mid_L 5 Frontal_Sup_Orb_L 9 Frontal_Mid_Orb_L
8 Frontal_Mid_R 29 Insula_L 36 Cingulum_Post_R 12 Frontal_Inf_Oper_R 14 Frontal_Inf_Tri_R
10 Frontal_Mid_Orb_R 39 ParaHippocampal_L 43 Calcarine_L 37 Hippocampus_L 37 Hippocampus_L
12 Frontal_Inf_Oper_R 42 Amygdala_R 45 Cuneus_L 39 ParaHippocampal_L 38 Hippocampus_R
34 Cingulum_Mid_R 49 Occipital_Sup_L 46 Cuneus_R, 40 ParaHippocampal_R 43 Calcarine_L
39 ParaHippocampal_L 51 Occipital_Mid_L 50 Occipital_Sup_R 43 Calcarine_L 48 Lingual_R
40 Parahippocampal_R 52 Occipital_Mid_R 56 Fusiform_R 47 Lingual_L 52 Occipital_Mid_R
50 Occipital_Sup_R 58 Postcentral_R 57 Postcentral_L 50 Occipital_Sup_R 67 Precuneus_L
57 Postcentral_L 60 Parietal_sup_R 74 Putamen_R 54 Occipital_Inf_R 68 Precuneus_R
78 Thalamus_R 67 Precuneus_L 90 Temporal_Inf_R 55 Fusiform_L 74 Putamen_R
TABLE II: Indices and names of regions visualized by MP-GAN using the AAL template.

The performance of MP-GAN to visualize the subtle lesions in the hippocampus is further investigated. The class-discriminative maps of the hippocampus in the sagittal view are visualized in Fig. 10. Specifically, the following 4 neighborhood evaluation groups are further explored: (a) NC vs. SMC; (b) SMC vs. EMCI; (c) EMCI vs. LMCI; (d) LMCI vs. AD. From Fig. 10, it can be observed that the zoomed regions preserve more details in the hippocampus. In particular, in the earlier stages of AD such as (a) NC vs. SMC and (b) SMC vs. EMCI, the visualized lesions are extremely subtle and scattered around the boundary of the hippocampus. In the later stages of AD such as (c) EMCI vs. LMCI and (d) LMCI vs. AD, the visualized lesions are accumulated at the core region of the hippocampus. Furthermore, Fig. 10(a) to Fig. 10(d) reflect the shape change and atrophy of the hippocampus qualitatively as the progressive deterioration from SMC to AD. It has already been validated by the previous studies [37] that the hippocampus is significant for identifying biomarkers in clinical practice. Although the volume loss and shape change of the hippocampus can not be quantitatively measured in this work, the visualized lesions of the hippocampus are beneficial for identifying the biomarkers in future work. Based on these visualized lesions in Fig. 10, the existing biomarkers such as Brain boundary shift integral (BBSI)[12], Scoring by Non-local Image Patch Estimator(SNIPE)[9], and other grading biomarkers[50] can be computed. Furthermore, new potential biomarkers reflecting the shape change and brain atrophy might be discovered based on these visualized lesions in the hippocampus in future work.

Iv-D Quantitative Analysis

In this section, the following 4 metrics are computed to assess visual quality. (1) Normalized Cross-Correlation (NCC). NCC [5]

is calculated between the ground-truth maps and the predicted class-discriminative maps. The higher the NCC, the more correlation between ground-truth maps and the predicted class-discriminative maps. For Integrated Gradients, Guided Backprop, and CAM, the visualized heatmaps for predicting positive class are utilized to calculate the NCC; (2) Peak Signal-to-Noise Ratio (PSNR). PSNR

[18] is also calculated between the ground-truth maps and the predicted class-discriminative maps on the test data set. Similar to NCC, the higher the PSNR, the closer between ground-truth maps and the predicted class-discriminative maps; (3) Structural Similarity Index Measure (SSIM) [60]. Different from NCC and PSNR, SSIM in each iteration is calculated between synthetic images and real images on the validation dataset. Higher SSIM indicates better reconstructed MR image quality. By computing SSIM in each iteration, the convergency of the model is further validated; (4) Classification metrics [57] such as AUC, ACC, Sensitivity, and Specificity for data augmentation. Note that the purpose of SSIM and the classification metrics is to demonstrate that the proposed MP-GAN can generate images close to real distribution, thus it validates that MP-GAN can capture salient global features in class-discriminative maps. For NCC and PSNR, the 4 existing methods are compared. For SSIM, only GAN is compared since the other 3 methods are based on classification. Similarly, for the classification metrics, only GAN is compared since the classification performance is based on synthetic data augmentation by the proposed MP-GAN and GAN.

Fig. 11: Box-plots of NCC for different models.

The NCC results shown in Fig. 11 are mostly consistent with the qualitative results shown from Fig. 4 to Fig. 7. The proposed MP-GAN achieves significantly higher NCC than the other 4 existing methods. It indicates that the distribution of class-discriminative maps generated by MP-GAN is the closest to ground-truth maps. The three methods based on classification (Integrated Gradients, Guided Backprop, and CAM) achieve a low NCC score due to their exclusive focus on local features. GAN performs better than 3 classification-based feature visualization methods for NC vs. SMC, NC vs. LMCI, and NC vs. AD. This implies that the GAN architecture can capture global features, which alleviates the limitations of feature visualization methods based on classification. Above all, the proposed MP-GAN achieves the highest correlation scores compared with the other 4 existing methods in all 4 evaluation groups.

Fig. 12: Box-plots of PSNR for different models.

From Fig. 12, it can be seen that the proposed MP-GAN achieves the best PSNR compared with the other 4 existing methods. This is also consistent with NCC results in Fig. 11 and the qualitative results shown from Fig. 4 to Fig. 7. The class-discriminative maps visualized by MP-GAN are closer to ground-truth. This is because MP-GAN benefits from the multidirectional mapping mechanism and the hybrid loss function. Meanwhile, MP-GAN can be trained on MR images of all classes with only one model. In this manner, the common features unrelated to the disease can be reused, thus all salient global features can be captured in class-discriminative maps for different Alzheimer’s stages.

Fig. 13: Convergence curves for NC vs. SMC.
Fig. 14: Convergence curves for NC vs. EMCI.
Fig. 15: Convergence curves for NC vs. LMCI.
Fig. 16: Convergence curves for NC vs. AD.
Fig. 17: The classification results of synthetic data augmentation for NC vs. SMC.
Fig. 18: The classification results of synthetic data augmentation for NC vs. EMCI.
Fig. 19: The classification results of synthetic data augmentation for NC vs. LMCI.
Fig. 20: The classification results of synthetic data augmentation for NC vs. AD.

Furthermore, the generation diversity with SSIM is evaluated in each iteration on the validation dataset. The convergence curves of the proposed MP-GAN and GAN are given for 4 evaluation groups: (1)NC vs. SMC, (2)NC vs. EMCI, (3)NC vs. LMCI, and (4)NC vs. AD respectively. From Fig. 13 to Fig. 16, it can be observed that the proposed MP-GAN converges faster than GAN. Meanwhile, MP-GAN performs stably in all 4 evaluation groups. On the other hand, the training of GAN is extremely unstable for NC vs. LMCI, and it can not converge for NC vs. EMCI and NC vs. AD. Again, these results are consistent with the NCC score in Fig. 11. The NCC score of GAN is low for NC vs. EMCI in Fig. 11, because GAN can’t converge for NC vs. EMCI as shown in Fig. 14. These results also indicate that the proposed MP-GAN can generate diverse MR images close to the real distribution.

The objective of the synthetic data augmentation is to demonstrate the learned class-discriminative map have captured all subtle morphological feature for different stages of AD. If the classification performance is improved, this indicates that the learned class-discriminative maps have captured all subtle morphological features for different stages of AD progression. Thus the synthetic MR images produced by the class-discriminative maps can be classified as the corresponding class correctly. By conducting this experiment, the efficacy of the MP-GAN is corroborated. More specifically, the CNN classifier is trained using synthetic data augmentation. More specifically, the 100 synthesized MR images of each class by MP-GAN and GAN are added to the original training set to form two new augmented training sets separately. Then the CNN model is trained on the two new augmented training sets separately for each evaluation group. During the test stage, the same test set of real MR images are used. From Fig. 17 to Fig. 20, it can be seen that adding synthesized samples by the proposed MP-GAN achieves better classification performance in terms of AUC, accuracy, specificity, and sensitivity. Overall, the synthetic data samples generated by MP-GAN can add additional variability to the original training set, which in turn leads to better performance. This implies that the synthesized MR images generated by MP-GAN not only provide meaningful visualizations but also capture the discriminative features for AD analysis. The proposed MP-GAN can be used as an effective data augmentation method.

V Discussion

Although the extensive experiments demonstrate the superiority of the proposed MP-GAN. However, MP-GAN has two limitations: (1) The hyperparameters are tuned empirically for the best performance. The optimal value of the hyperparameters depends on network architecture and data. There is no straightforward way to find optimal hyperparameters in advance. (2) The conventional GAN has several common failure modes, such as training instability and mode Collapse. The proposed MP-GAN is designed as a three-player cooperative game instead of the conventional two-player competition game by introducing the auxiliary classifier network based on generator and discriminator. The specific architecture design and the proposed hybrid loss can make the training process more stable. However, mode collapse, which is a common issue of the GAN model, still might happen even MP-GAN has shown stable training performance. In future works, how to solve the mode collapse issue is the direction to further improve the robustness of the proposed MP-GAN.

Vi Conclusion

In this paper, a novel MP-GAN is proposed to visualize the morphological features indicating the severity of AD in whole-brain MR images. By introducing a novel multidirectional mapping mechanism into the model, MP-GAN can capture the salient global features efficiently. Thus, by utilizing the class-discriminative map from the generator, the proposed model can clearly delineate the subtle lesions via MR image transformations between the source domain and the target domain. Besides, by integrating the adversarial loss, classification loss, cycle consistency loss, and L1 penalty, a single generator in MP-GAN can learn the class-discriminative maps for multiple-classes. Experimental results on the public ADNI dataset have demonstrated that MP-GAN can visualize multiple lesions affected by the progression of AD accurately. Furthermore, MP-GAN may visualize some new disease-related regions that have not been investigated yet. This can be studied further to discover potential new AD biomarkers in future work.

Acknowledgment

This work was supported by the National Natural Science Foundations of China under Grants 62172403 and 61872351, the International Science and Technology Cooperation Projects of Guangdong under Grant 2019A050510030, the Distinguished Young Scholars Fund of Guangdong under Grant 2021B1515020019, the Excellent Young Scholars of Shenzhen under Grant RCYX20200714114641211 and Shenzhen Key Basic Research Projects under Grant JCYJ20200109115641762 and JCYJ20180507182506416, the HKRGC Grant Numbers: GRF 12200317, 12300218, 12300519 and 17201020, and the data support of ADNI.

References

  • [1] M. Ancona, E. Ceolini, A. C. Öztireli, and M. H. Gross (2017)

    A unified view of gradient-based attribution methods for deep neural networks

    .
    arXiv e-prints abs/1711.06104. Cited by: item 2.
  • [2] M. Arjovsky, S. Chintala, and L. Bottou (2017-06–11 Aug) Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning, D. Precup and Y. W. Teh (Eds.), Proceedings of Machine Learning Research, Vol. 70, pp. 214–223. Cited by: §II.
  • [3] A. BA (2014) Corpus callosum shape changes in early alzheimer’s disease: an mri study using the oasis brain database. Brain Struct Funct 219(1), pp. 343–352. External Links: Document Cited by: §IV-C.
  • [4] G. S. Babu and S. Suresh (2013)

    Sequential projection-based metacognitive learning in a radial basis function network for classification problems

    .
    IEEE Transactions on Neural Networks and Learning Systems 24 (2), pp. 194–206. External Links: Document Cited by: §I.
  • [5] C. F. Baumgartner, L. M. Koch, K. C. Tezcan, J. X. Ang, and E. Konukoglu (2017) Visual feature attribution using wasserstein gans.

    IEEE/CVF Conference on Computer Vision and Pattern Recognition

    , pp. 8309–8319.
    Cited by: §IV-C, §IV-D.
  • [6] M. Bohle, F. Eitel, M. Weygandt, and K. Ritter (2019) Layer-wise relevance propagation for explaining deep neural network decisions in mri-based alzheimer’s disease classification. Frontiers in Aging Neuroscience 11, pp. 194. External Links: ISSN 1663-4365 Cited by: item 2.
  • [7] M. W. Bondi, E. C. Edmonds, and D. P. Salmon (2017-10) Alzheimer’s disease: past, present, and future. Journal of the International Neuropsychological Society : JINS 23 (9-10), pp. 818—831. External Links: Document, ISSN 1355-6177 Cited by: §I.
  • [8] H. Braak and E. Braak (1991) Neuropathological stageing of alzheimer-related changes. Acta neuropathologica 82 (4), pp. 239—259. External Links: Document, ISSN 0001-6322 Cited by: §IV-C.
  • [9] P. Coupé, S. F. Eskildsen, J. V. Manjón, V. S. Fonov, and D. L. Collins (2012) Simultaneous segmentation and grading of anatomical structures for patient’s classification: application to alzheimer’s disease. NeuroImage 59 (4), pp. 3736 – 3747. External Links: ISSN 1053-8119 Cited by: §IV-C.
  • [10] B. C. Dickerson, A. Bakkour, D. H. Salat, E. Feczko, J. Pacheco, D. N. Greve, F. Grodstein, C. I. Wright, D. Blacker, H. D. Rosas, R. A. Sperling, A. Atri, J. H. Growdon, B. T. Hyman, J. C. Morris, B. Fischl, and R. L. Buckner (2008-07) The Cortical Signature of Alzheimer’s Disease: Regionally Specific Cortical Thinning Relates to Symptom Severity in Very Mild to Mild AD Dementia and is Detectable in Asymptomatic Amyloid-Positive Individuals. Cerebral Cortex 19 (3), pp. 497–510. External Links: ISSN 1047-3211, Document Cited by: §IV-C.
  • [11] C. Feng, A. Elazab, P. Yang, T. Wang, F. Zhou, H. Hu, X. Xiao, and B. Lei (2019) Deep learning framework for alzheimer’s disease diagnosis via 3d-cnn and fsbi-lstm. IEEE Access 7 (), pp. 63605–63618. Cited by: §IV-C.
  • [12] N. C. Fox and P. A. Freeborough (1997) Brain atrophy progression measured from registered serial mri: validation and application to alzheimer’s disease. JMRI 7 (6), pp. 1069–1075. Cited by: §IV-C.
  • [13] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio (2014) Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672–2680. Cited by: §I, §II.
  • [14] D. Gu (2017) 3D densely connected convolutional network for the recognition of human shopping actions. In http://dx.doi.org/10.20381/ruor-21013, Cited by: §III-B.
  • [15] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville (2017) Improved training of wasserstein gans. In NIPS, pp. 5769–5779. Cited by: §II.
  • [16] X. Hao, Y. Bao, Y. Guo, M. Yu, D. Zhang, S. L. Risacher, A. J. Saykin, X. Yao, and L. Shen (2020)

    Multi-modal neuroimaging feature selection with consistent metric constraint for diagnosis of alzheimer’s disease

    .
    Medical Image Analysis 60, pp. 101625. External Links: ISSN 1361-8415 Cited by: §I, §I, §IV-C.
  • [17] K. He, X. Zhang, S. Ren, and J. Sun (2016-06) Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §III-B.
  • [18] A. Horé and D. Ziou (2010) Image quality metrics: psnr vs. ssim. In 2010 20th International Conference on Pattern Recognition, Vol. , pp. 2366–2369. External Links: Document Cited by: §IV-D.
  • [19] S. Hu, B. Lei, S. Wang, Y. Wang, Z. Feng, and Y. Shen (2021) Bidirectional mapping generative adversarial networks for brain mr to pet synthesis. IEEE Transactions on Medical Imaging. Cited by: §I.
  • [20] S. Hu, Y. Shen, S. Wang, and B. Lei (2020) Brain mr to pet synthesis via bidirectional generative adversarial network. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 698–707. Cited by: §I.
  • [21] S. Hu, W. Yu, Z. Chen, and S. Wang (2020) Medical image reconstruction using generative adversarial network for alzheimer disease assessment with class-imbalance problem. In 2020 IEEE 6th International Conference on Computer and Communications (ICCC), pp. 1323–1327. Cited by: §I.
  • [22] S. Hu, J. Yuan, and S. Wang (2019) Cross-modality synthesis from mri to pet using adversarial u-net with different normalization. In 2019 International Conference on Medical Imaging Physics and Engineering (ICMIPE), pp. 1–5. Cited by: §I.
  • [23] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger (2017) Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Cited by: §III-B.
  • [24] S. Ioffe and C. Szegedy (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv e-prints abs/1502.03167. Cited by: §III-B.
  • [25] M. Jenkinson, P. Bannister, M. Brady, and S. Smith (2002) Improved optimization for the robust and accurate linear registration and motion correction of brain images. NeuroImage 17 (2), pp. 825 – 841. External Links: ISSN 1053-8119 Cited by: §IV-A.
  • [26] M. Jenkinson and S. Smith (2001) A global optimisation method for robust affine registration of brain images. Medical Image Analysis 5 (2), pp. 143 – 156. External Links: ISSN 1361-8415 Cited by: §IV-A.
  • [27] B. Jie, M. Liu, and D. Shen (2018) Integration of temporal and spatial properties of dynamic connectivity networks for automatic diagnosis of brain disease. Medical Image Analysis 47, pp. 81 – 94. External Links: ISSN 1361-8415 Cited by: §I, §II.
  • [28] N. M. Khan, N. Abraham, and M. Hon (2019) Transfer learning with intelligent training data selection for prediction of alzheimer’s disease. IEEE Access 7 (), pp. 72726–72735. Cited by: item 3.
  • [29] T. Kim, M. Cha, H. Kim, J. K. Lee, and J. Kim (2017) Learning to discover cross-domain relations with generative adversarial networks. arXiv e-prints abs/1703.05192. Cited by: §III-B.
  • [30] S. Korolev, A. Safiullin, M. Belyaev, and Y. Dodonova (2017) Residual and plain convolutional neural networks for 3d brain mri classification. In 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017), pp. 835–838. External Links: Document Cited by: item 1, §IV-C.
  • [31] B. Lei, E. Liang, M. Yang, P. Yang, F. Zhou, E. Tan, Y. Lei, C. Liu, T. Wang, X. Xiao, et al. (2022) Predicting clinical scores for alzheimer’s disease based on joint and deep learning. Expert Systems with Applications 187, pp. 115966. Cited by: §I.
  • [32] B. Lei, M. Yang, P. Yang, F. Zhou, W. Hou, W. Zou, X. Li, T. Wang, X. Xiao, and S. Wang (2020) Deep and joint learning of longitudinal data for alzheimer’s disease prediction. Pattern Recognition 102, pp. 107247. Cited by: §I.
  • [33] C. Lian, M. Liu, L. Wang, and D. Shen (2019-01) End-to-end dementia status prediction from brain mri using multi-task weakly-supervised attention network. In Medical Image Computing and Computer Assisted Intervention(MICCAI), pp. 158–167. External Links: Document Cited by: item 3.
  • [34] C. Lian, M. Liu, J. Zhang, and D. Shen (2018) Hierarchical fully convolutional network for joint atrophy localization and alzheimer’s disease diagnosis using structural mri. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 50–60. Cited by: §I, §II.
  • [35] A. Mahendran and A. Vedaldi (2016) Salient deconvolutional networks. In Computer Vision – ECCV 2016, pp. 120–135. External Links: ISBN 978-3-319-46466-4 Cited by: item 2.
  • [36] N. Mammone, C. Ieracitano, H. Adeli, A. Bramanti, and F. C. Morabito (2018)

    Permutation jaccard distance-based hierarchical clustering to estimate eeg network density modifications in mci subjects

    .
    IEEE Transactions on Neural Networks and Learning Systems 29 (10), pp. 5122–5135. External Links: Document Cited by: §I.
  • [37] F. Márquez and M. A. Yassa (2019-06) Neuroimaging biomarkers for alzheimer’s disease. Molecular neurodegeneration 14 (1), pp. 21. External Links: Document, ISSN 1750-1326 Cited by: §IV-C.
  • [38] L. Mo and S. Wang (2009) A variational approach to nonlinear two-point boundary value problems. Nonlinear Analysis: Theory, Methods & Applications 71 (12), pp. e834–e838. Cited by: §I.
  • [39] L. Nie, L. Zhang, L. Meng, X. Song, X. Chang, and X. Li (2017) Modeling disease progression via multisource multitask learners: a case study with alzheimer’s disease. IEEE Transactions on Neural Networks and Learning Systems 28 (7), pp. 1508–1519. External Links: Document Cited by: §IV-C.
  • [40] E. Nigri, N. Ziviani, F. Cappabianco, A. Antunes, and A. Veloso (2020) Explainable deep cnns for mri-based diagnosis of alzheimer’s disease. arXiv e-prints. Cited by: item 1.
  • [41] J. M. Rondina, L. K. Ferreira, F. L. de Souza Duran, R. Kubo, C. R. Ono, C. C. Leite, J. Smid, R. Nitrini, C. A. Buchpiguel, and G. F. Busatto (2018) Selecting the most relevant brain regions to discriminate alzheimer’s disease patients from healthy controls using multiple kernel learning: a comparison across functional and structural imaging modalities and atlases. NeuroImage: Clinical 17, pp. 628 – 641. External Links: ISSN 2213-1582 Cited by: §IV-C.
  • [42] S. Sarraf and G. Tofighi (2016-07) Classification of Alzheimer’s Disease Structural MRI Data by Deep Learning Convolutional Neural Networks. arXiv e-prints, pp. arXiv:1607.06583. Cited by: item 3.
  • [43] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra (2017-10) Grad-cam: visual explanations from deep networks via gradient-based localization. In The IEEE International Conference on Computer Vision (ICCV), Cited by: item 2.
  • [44] Y. Shi, H. -I. Suk, Y. Gao, S. -W. Lee, and D. Shen (2020) Leveraging coupled interaction for multimodal alzheimer’s disease diagnosis. IEEE Transactions on Neural Networks and Learning Systems 31 (1), pp. 186–200. External Links: Document Cited by: §I.
  • [45] K. Simonyan, A. Vedaldi, and A. Zisserman (2014) Deep inside convolutional networks: visualising image classification models and saliency maps. edition, , Vol. , , pp. 1–8. Cited by: item 2.
  • [46] S. M. Smith, M. Jenkinson, M. W. Woolrich, C. F. Beckmann, T. E.J. Behrens, H. Johansen-Berg, P. R. Bannister, M. [. Luca], I. Drobnjak, D. E. Flitney, R. K. Niazy, J. Saunders, J. Vickers, Y. Zhang, N. [. Stefano], J. M. Brady, and P. M. Matthews (2004) Advances in functional and structural mr image analysis and implementation as fsl. NeuroImage 23, pp. S208 – S219. Cited by: §IV-A.
  • [47] S. M. Smith (2002-11) Fast robust automated brain extraction. Human brain mapping 17 (3), pp. 143—155. External Links: Document, ISSN 1065-9471 Cited by: §IV-A.
  • [48] J.T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller (2015) Striving for simplicity: the all convolutional net. In ICLR, Cited by: item 2, §IV-C.
  • [49] M. Sundararajan, A. Taly, and Q. Yan (2017) Axiomatic attribution for deep networks. arXiv e-prints abs/1703.01365. Cited by: item 2, §IV-C.
  • [50] T. Tong, Q. Gao, R. Guerrero, C. Ledig, L. Chen, D. Rueckert, and A. D. N. Initiative (2017) A novel grading biomarker for the prediction of conversion from mild cognitive impairment to alzheimer’s disease. IEEE Transactions on Biomedical Engineering 64 (1), pp. 155–165. External Links: Document Cited by: §IV-C.
  • [51] D. Ulyanov, A. Vedaldi, and V. S. Lempitsky (2016) Instance normalization: the missing ingredient for fast stylization. arXiv e-prints abs/1607.08022. Cited by: §III-B.
  • [52] M. Wang, D. Zhang, D. Shen, and M. Liu (2019) Multi-task exclusive relationship learning for alzheimer’s disease progression prediction with longitudinal data. Medical Image Analysis 53, pp. 111 – 122. External Links: ISSN 1361-8415 Cited by: §I, §II.
  • [53] S. Wang and J. He (2008) Variational iteration method for a nonlinear reaction-diffusion process. International Journal of Chemical Reactor Engineering 6 (1). Cited by: §I.
  • [54] S. Wang (2009) A variational approach to nonlinear two-point boundary value problems. Computers & Mathematics with Applications 58 (11-12), pp. 2452–2455. Cited by: §I.
  • [55] S. Wang, Y. Shen, W. Chen, T. Xiao, and J. Hu (2017) Automatic recognition of mild cognitive impairment from mri images using expedited convolutional neural networks. In International Conference on Artificial Neural Networks, pp. 373–380. Cited by: §I.
  • [56] S. Wang, Y. Shen, D. Zeng, and Y. Hu (2018) Bone age assessment using convolutional neural networks. In 2018 International Conference on Artificial Intelligence and Big Data (ICAIBD), pp. 175–178. Cited by: §I.
  • [57] S. Wang, H. Wang, Y. Shen, and X. Wang (2018) Automatic recognition of mild cognitive impairment and alzheimers disease using ensemble based 3d densely connected convolutional networks. In 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), Vol. , pp. 517–523. External Links: Document Cited by: §I, §IV-D.
  • [58] S. Wang, X. Wang, Y. Hu, Y. Shen, Z. Yang, M. Gan, and B. Lei (2021) Diabetic retinopathy diagnosis using multichannel generative adversarial network with semisupervision. IEEE Transactions on Automation Science and Engineering 18 (2), pp. 574–585. External Links: Document Cited by: §II.
  • [59] S. Wang, X. Wang, Y. Shen, B. He, X. Zhao, P. W. Cheung, J. P. Y. Cheung, K. D. Luk, and Y. Hu (2020) An ensemble-based densely-connected deep learning system for assessment of skeletal maturity. IEEE Transactions on Systems, Man, and Cybernetics: Systems. Cited by: §I.
  • [60] Z. Wang, E. P. Simoncelli, and A. C. Bovik (2003) Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems Computers, Vol. 2, pp. 1398–1402. External Links: Document Cited by: §IV-D.
  • [61] M. W. Woolrich, S. Jbabdi, B. Patenaude, M. Chappell, S. Makni, T. Behrens, C. Beckmann, M. Jenkinson, and S. M. Smith (2009) Bayesian analysis of neuroimaging data in fsl. NeuroImage 45, pp. S173 – S186. Cited by: §IV-A.
  • [62] X. Yi, E. Walia, and P. Babyn (2018-09) Generative Adversarial Network in Medical Imaging: A Review. arXiv e-prints. External Links: 1809.07294 Cited by: §I, §II.
  • [63] J. Yosinski, J. Clune, A. M. Nguyen, T. J. Fuchs, and H. Lipson (2015) Understanding neural networks through deep visualization. arXiv e-prints abs/1506.06579. Cited by: item 2.
  • [64] S. Yu, S. Wang, X. Xiao, J. Cao, G. Yue, D. Liu, T. Wang, Y. Xu, and B. Lei (2020) Multi-scale enhanced graph convolutional network for early mild cognitive impairment detection. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 228–237. Cited by: §I.
  • [65] W. Yu, B. Lei, M. K. Ng, A. C. Cheung, Y. Shen, and S. Wang (2021) Tensorizing gan with high-order pooling for alzheimer’s disease assessment. IEEE Transactions on Neural Networks and Learning Systems (), pp. 1–15. External Links: Document Cited by: §II.
  • [66] M. D. Zeiler and R. Fergus (2014) Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014, pp. 818–833. External Links: ISBN 978-3-319-10590-1 Cited by: item 1.
  • [67] Y. Zhang, Z. Dong, P. Phillips, S. Wang, G. Ji, J. Yang, and T. Yuan (2015) Detection of subjects and brain regions related to alzheimer’s disease using 3d mri scans based on eigenbrain and machine learning. Frontiers in Computational Neuroscience 9, pp. 66. External Links: Document, ISSN 1662-5188 Cited by: §IV-C.
  • [68] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba (2016-06)

    Learning deep features for discriminative localization

    .
    In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: item 3, §IV-C.
  • [69] J. Zhu, T. Park, P. Isola, and A. A. Efros (2017-10) Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Cited by: §II, §III-B.