1 Introduction
Alzheimer’s disease (AD) is an irreversible, chronic neurodegenerative disease, and is the main reason for dementia among aged people [1]. Those people who suffered from AD will gradually lose cognitive function such as remembering or thinking, and eventually lose the ability to perform daily activates [2, 3]. During the past few decades, there are two methods that are widely applied to diagnose AD or other neurodegenerative diseases. One is restingstate functional magnetic resonance imaging(rsfMRI) [4],which is based on bloodoxygenleveldependent(BOLD)signals; the other is Diffusion Tensor Imaging(DTI) [5]
, which uses the white matter fiber tractography. But unfortunately, the cause and mechanism of AD are still not entirely clear. To overcome such barriers, new tools should be implemented. One of the modern approaches is the analysis of the brain network. Brain network is a representation of connectivity maps between brain regions(defined in an anatomical parcellation, or brain atlas), which can give essential insights in studying brain science. Specifically, Brain network provides enormous information about global and local features of neuronal network architecture, which can evaluate the disease status, identify crucial brain regions of AD, and reveal the mechanism of AD. In terms of mathematics, a brain network is equivalent to a weighted undirected graph, where each node in the graph corresponds to a brain region, and the value of edge between two different nodes represents the strength of connectivity of the corresponding brain regions. As the development of deep learning technology for neuroimaging data provides powerful tools to compute and analyze the brain network, many studies
[6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16] have exploited deep learning models to obtain ADrelated features from brain network. Recent studies[17, 18, 19, 20, 21, 22, 23]have shown that the combination of multimodal neuroimging data can discover complementary information of brain network, which is beneficial to improve the deep feature representation. Therefore, designing an effective multimodal fusion model for computing brain network has become a hot topic. However, most existing multimodal fusion models directly use neural fiber tractography (i.e., structural connectivity, short for SC) to determine the edges between brain regions and use the signal of brain activities (i.e., BOLD) to characterize the nodes feature. Due to the heterogeneity and complexity between BOLD signals and fiber tractography, such methods can not sufficiently take advantage of the complementary information. Moreover, the previous studies have demonstrated that brain cognitive mechanisms involve multiple coactivated brain regions (i.e., neural circuit) interactions rather than single pairwise interactions
[24, 25]. To overcome the above problem, some researchers began to use hypergraph to characterize brain network [26]. Hypergraphs are a generalization of graphs, which are increasingly important in data science thanks to the development of combinatorial mathematics and computer science. The most significant difference between hypergraphs and graphs is that each hyperedge in a hypergraph can contain multiple nodes. This feature of hypergraphs is a natural superiority to analyze the neural circuit in brain network. On the other hand, Generative Adversarial Network (GAN)
[27, 28, 29, 30, 31, 32]is nowadays the dominant method for processing multimodal medical images. GAN has a strong ability to deal with unsupervised learning tasks. Motivated by this advantage, we devise a hypergraph Generative Adversarial Networks (HGGAN) to characterize the multimodal connectivity of brain network based on the hypergraph theory for AD analysis.
In this paper, we propose a HGGAN to generate multimodal connectivity from rsfMRI and DTI. We design the generator of HGGAN using the Interactive Hyperedge Neurons(IHEN) module, which is an improvement of recent work HNHN [33]. Compared with classical CNNbased GAN that can only operate on Euclidean data, our proposed model can learn intrinsic relationships and complementary information based on hypergraph structure multimodal imaging data.
2 Method
Figure 1 shows the proposed framework for multimodal connectivity generation. Specifically, we first extract the BOLD time series of each brain region (90 regions in total) from rsfMRI data by using AAL atlas [34]. We constructed SC from DTI data by using PANDA [36]. We use BOLD to: (i) represent the features of nodes and (ii) calculate the incidence matrix of the hypergraph through the OHGH algorithm. And we use SC to calculate the features of hyperedges. In this way the features of nodes, the incidence matrix, and the features of hyperedges were fed into the generator to generate the multimodal connectivity. Simultaneously, we utilize GRETNA [35] to obtain functional connectivity (FC), and FC is used as real samples to train the discriminator.
2.1 Data and Preprocessing
We select a total of 219 subjects from the ADNI database. The composition of our dataset is given as follows: Our dataset includes 18 AD female patients and 32 AD male patients; 13 late mild cognitive impairment (LMCI) female patients and 12 LMCI male patients; 24 early MCI (EMCI) female patients and 45 EMCI male patients; 43 normal control (NC) female and 32 male NC. The average age of AD LMCI, EMCI and NC is 75.3, 74.9, 75.8 and 74.0, respectively.
Preprocessing of rsfMRI data uses the DPARSF [37] toolbox and the GRETNA toolbox based on the data analysis software statistical parameter mapping (SPM12) [38]. First we use DPARSF to preprocess the initial DICOM format of rsfMRI data into NIFTI format data. We apply the standard steps for rsfMRI data preprocessing, including the discarding of the first 20 volumes, head motion correction, spatial normalization, and Gaussian smoothing in this stage. Then we use the AAL atlas to divide brain space into 90 brain regionsofinterest (ROIs). Finally, we send these NIFTI format data into the brain network analysis toolbox GRETNA to extract the BOLD time series corresponding to ROIs. We utilized DPARSF to obtain BOLD time series and we utilized GRETNA to obtain FC matrix. Also by using the AAL atlas, Preprocessing of DTI data uses the PANDA toolbox so that the number of white matter fiber tractography can be regarded as strength of physical connections in the SC matrix.
2.2 Hypergraph and Optimal Hypergraph Homomorphism Algorithm
We first introduce the basic concepts and notations of hypergraphs. We define a hypergraph to consist of a set of nodes and set of hyperedges, where each hyperedge itself is a set of nodes. Let and . We label the nodes as for , and the hyperedges as for . The incidence matrix of hypergraph is denoted by
In this paper, we regard each ROI as a node. We apply the Dynamic Hypergraph Construction algorithm (DHC) [39] to calculate the initial hypergraphs for every subject. However, each initial hypergraph only focuses on representing multiple coactivated ROIs information for the corresponding subject. Therefore, if we directly send initial hypergraphs to the generator to obtain multimodal connectivity, the robustness of the generation result cannot be ensured. To overcome this shortage, we proposed a novel Optimal Hypergraph Homomorphism algorithm(OHGH). OHGH can calculate the hypergraph which have optimal homomorphism with respect to the initial hypergraphs . Before introducing OHGH, we first define the similarity on hypergraphs. Given two hypergraphs and with the same nodes and , the similarity between is defined by
where
Now we can give OHGH as follows
With the optimal homomorphism hypergraph , we construct the hypergraphs for every subject by concatenating and (i.e., ). Furthermore, we obtain the incidence matrices based on .
2.3 Generator and Interactive Hyperedge Neurons Module
For each subject, say for the th subject, we use the BOLD time series of the th subject to represent the nodes features and we rewrite it as (where is the number of nodes, is the length of the BOLD time series). Meanwhile, we calculate the hyperedges features by using the SC matrix and the incidence matrix of the th subject (where is the number of hyperedges). Specifically, it is computed in the following way
The proposed generator is composed of multilayer Interactive Hyperedge Neurons modules (IHEN). The definition of IHEN is given as follows
where and are the th layer features of nodes and hyperedges, and are the th layer weight matrices of nodes and hyperedges,
is hyperparameters,
We compute the hyperedges weights and the hyperedgeindependent nodes weights by using and as follows:
where is the weight of the th node with respect to the th hyperedge for the th subject and is the weight of the the th hyperedge for the th subject. From this, we obtain the multimodal connectivity matrix for the th subject by the formulas
(1) 
Combining the multimodal connectivity matrix with the nodes features , we give the nodes correlation coefficients as follows:
(2) 
2.4 Discriminator and Loss Function
The discriminator is a normal MLP architecture, which can compare the difference of distributions between the FC matrix and the multimodal connectivity matrix by using random walk. Fixing an arbitrary node , we take as the starting point for the random walk. The termination condition of the random walk is that the point comes back to the position of the previous step at some step of random walk. For example, a random walk having steps and eventually stopping at the means that the path satisfies , and we say the is the endpoint of the path. For the FC matrix of the
th subject, the probability of walking through the path
is given as follows:Let be the set of paths with the starting point at and the endpoint at , then the conditional probability about is given as follows
Similarly, for the multimodal connectivity matrix of the th subject, the probability of walking through the path is given by
and conditional probability about is that
The loss function of discriminator is formulated by
where represents the total number of subject in the training dataset.
The loss function of the generators depends on the feedback of discriminator, which is formulated by
3 Experiments
In order to evaluate the performance of the multimodal connectivity matrix(MC) generated by our proposed method, six binary classification experiments including AD vs. NC, AD vs. EMCI , AD vs. LMCI, EMCI vs. NC , LMCI vs. NC, and EMCI vs. LMCI are designed to validate the prediction performance. We use prediction accuracy (ACC), sensitivity (SEN), specificity (SPE) to measure the results of our experiments.
Classifier  Connectivity Matrix  AD vs. NC  AD vs. EMCI  AD vs. LMCI  

ACC  SEN  SPE  ACC  SEN  SPE  ACC  SEN  SPE  
MLP  SC  79.06  52.94  96.15  78.04  88.23  65.38  60.00  76.47  25.00 
FC  74.41  52.94  88.46  68.29  64.70  65.38  56.00  64.70  37.50  
MC(Ours)  81.39  64.70  92.30  80.48  76.47  76.92  64.00  58.82  75.00  
SVM  SC  83.72  70.58  92.30  80.48  64.70  84.61  64.00  64.70  62.50 
FC  79.06  70.58  84.61  73.17  52.94  80.76  56.00  52.94  62.50  
MC(Ours)  86.04  82.35  88.46  82.92  76.47  80.76  68.00  70.58  62.50  
RF  SC  81.39  70.58  88.46  78.04  64.70  80.76  60.00  70.58  37.50 
FC  76.74  70.58  80.76  73.17  58.82  76.92  60.00  70.58  37.50  
MC(Ours)  86.04  88.23  84.61  82.92  82.35  76.92  64.00  88.23  12.50  
GCN  SC  86.04  94.11  80.76  85.36  94.11  73.07  68.00  76.47  50.00 
FC  83.72  94.11  76.92  80.48  70.58  80.76  64.00  76.47  37.50  
MC(Ours)  93.02  94.11  92.30  90.24  82.35  88.46  72.00  82.35  50.00 
Classifier  Connectivity Matrix  EMCI vs. NC  LMCI vs. NC  EMCI vs. LMCI  

ACC  SEN  SPE  ACC  SEN  SPE  ACC  SEN  SPE  
MLP  SC  66.00  62.50  69.23  76.47  50.00  84.61  68.75  79.16  37.50 
FC  64.00  70.83  57.69  73.52  75.00  73.07  71.87  75.00  62.50  
MC(Ours)  70.00  75.00  65.38  79.41  87.50  76.92  75.00  79.16  62.50  
SVM  SC  68.00  66.66  69.23  82.35  62.50  88.46  71.87  83.33  37.50 
FC  70.00  70.83  69.23  82.35  75.00  84.61  71.87  87.50  25.00  
MC(Ours)  72.00  83.33  61.53  85.29  87.50  84.61  78.12  87.50  50.00  
RF  SC  68.00  58.33  76.92  76.47  62.50  80.76  71.87  75.00  62.50 
FC  68.00  62.50  73.07  79.41  62.50  84.61  75.00  83.33  50.00  
MC(Ours)  74.00  66.66  80.76  82.35  87.50  80.76  81.25  79.16  87.50  
GCN  SC  72.00  70.83  73.07  85.29  87.50  84.61  78.12  83.33  62.50 
FC  74.00  75.00  73.07  79.41  75.00  80.76  81.25  91.66  50.00  
MC(Ours)  80.00  79.16  80.76  91.17  87.50  92.30  90.62  91.66  87.50 
Table 1 and Table 2 summarize the prediction results in different classifiers by using structural connectivity matrix (SC), functional connectivity matrix (FC), and our proposed multimodal connectivity matrix (MC) in eq.(1). Classifiers include Multiple Layer Perception (MLP) [40]
, Support Vector Machine (SVM)
[41], Random Forest (RF)
[42], and Graph Convolution Networks (GCN) [43]. In detail, the parameters of MLP set as follows: 3layers with 16, 16, 2 neurons, ReLU activation, 0.001 learning rate. The parameters of SVM set as follows: Gaussian Kernel, 0.15 kernel coefficient. The parameters of RF set as follows: 400 trees, 3 maximum depth. The parameters of GCN parameters set as follows: 3 Chebyshev graph convolutional layers, 0.1 dropout rate, 0.0001 learning rate. The dataset is randomly split into 65% for training and 35% for testing. All the models are randomly initialized for 5 times. The result shows that the classification accuracy is significantly improved by using the proposed MC matrix, which means the complementary information of different modal is successfully extracted from the proposed method.
We compare the average correlation coefficient in eq.(2) between each corresponding node of the AD group and the NC group. The result are shown in Fig 3. We choose the top 7 nodes in Fig 3 (c), the IDs of these node are 37,38,39,41,83,84, and 87, which represent Hippocampus_L, Hippocampus_R, ParaHippocampal_L, Amygdala_L, Temporal_Pole_Sup_L, Temporal_Pole_Sup_R, and Temporal_Pole_ Mid_L. We can see that these brain regions are mainly concentrated on the memory and reasoning areas, which are highly related to the AD according to the clinical studies [44]. Fig 4 and Fig 5 show the connection relationship of these top 7 brain regions by using MC matrix. We display the connection relationship graph of brain regions from the perspective of the sagittal plane view, axial plane view, and coronal plane view.
4 Conclution
In this paper, we proposed a novel hypergraph generative adversarial network (HGGAN) to generate individual multimodal connectivity matrix by using corresponding rsfMRI and DTI data. We designed the Interactive Hyperedge Neurons module such that the generators can efficiently capture the complex relationship between rsfMRI and DTI. Moreover, we proposed the Optimal Hypergraph Homomorphism algorithm to construct hypergraph structure data, which significantly improved the robustness of the generation results. The analyses of the experimental results proved that the proposed method successfully extracted the interrelated hidden structures and complementary information from different modal data. Although this paper only focuses on AD, it is worth mention that the proposed model can be easily extended to other neurodegenerative disease.
Acknowledgement.
This work was supported by the National Natural Science Foundations of China under Grant 61872351, the International Science and Technology Cooperation Projects of Guangdong under Grant 2019A050510030, the Distinguished Young Scholars Fund of Guangdong under Grant 2021B1515020019, the Excellent Young Scholars of Shenzhen under Grant RCYX20200714114641211 and Shenzhen Key Basic Research Project under Grant JCYJ20200109115641762.
References
 [1] Dadar, M., Pascoal, T.A., Manitsirikul, S., Fonov, V.S., et al.: Validation of a Regression Technique for Segmentation of White Matter Hyperintensities in Alzheimer’s Disease. IEEE Trans. Med. Imaging 36(8), pp. 1758–1768 (2017)
 [2] Association, A.s.: Alzheimer’s disease facts and figures. Alzheimer’s Dement 14, pp. 367–429 (2018)

[3]
Wang, S., Wang, H., Shen, Y., Wang, X.: Automatic recognition of mild cognitive impairment and Alzheimer’s disease using ensemble based 3d densely connected convolutional networks, 17th IEEE International Conference on Machine Learning and Applications (2018)
 [4] Huettel, S.A., Song, A.W., McCarthy, G.: Functional Magnetic Resonance Imaging. Sinauer Associates, Sunderland, MA (2004)
 [5] Westlye, L., et al.: Lifespan changes of the human brain white matter: diffusion tensor imaging (DTI) and volumetry. Cereb. Cortex 20, pp. 2055–2068 (2010). https:// doi.org/10.1093/cercor/bhp280
 [6] Wang, S., Hu, Y., Shen, Y., Li, H.: Classification of diffusion tensor metrics for the diagnosis of a myelopathic cord using machine learning. International journal of neural systems. 28 (02) , 1750036 (2018)

[7]
Wang, S., Shen, Y., Shi, C., Yin, P., Wang, Z., Cheung, P., Cheung, J.: Skeletal maturity recognition using a fully automated system with convolutional neural networks. IEEE Access 6, pp. 2997929993, (2018)

[8]
Wang, S., Shen, Y., Zeng, D., Hu, Y.: Bone age assessment using convolutional neural networks. International Conference on Artificial Intelligence and Big Data (ICAIBD), pp. 175–178, (2018)

[9]
Zeng, D., Wang, S., Shen, Y., Shi, C.: A GAbased feature selection and parameter optimization for support tucker machine. Procedia computer science,
111, pp. 17–23, (2017)  [10] Jeon, E., Kang, E., Lee, J., Lee, J., Kam, T., Suk, H.: Enriched representation learning in restingstate fMRI for early MCI diagnosis. In: Martel A.L. et al. (eds) MICCAI 2020, LNCS, vol. 12267, pp. 397–406. Springer, Cham (2020). 10.1007/9783030597283_39
 [11] Wang, S., Li, X., Cui, J., Li, H., Luk, K., Hu, Y.: Prediction of myelopathic level in cervical spondylotic myelopathy using diffusion tensor imaging. Journal of Magnetic Resonance Imaging 41 (6), pp. 1682–1688, (2015)
 [12] Mo, L., Wang, S.: A variational approach to nonlinear twopoint boundary value problems. Nonlinear Analysis: Theory, Methods & Applications, 71 (12) 834–838 (2009)
 [13] Wang, S.: A variational approach to nonlinear twopoint boundary value problems. Computers & Mathematics with Applications, 58 (1112) 2452–2455 (2009)
 [14] Wang, S., He, J.: Variational iteration method for a nonlinear reactiondiffusion process. International Journal of Chemical Reactor Engineering, 6 (1) (2008)
 [15] Wang, S., et al.: An EnsembleBased DenselyConnected Deep Learning System for Assessment of Skeletal Maturity. IEEE Transactions on Systems, Man, and Cybernetics: Systems, (2020)
 [16] Wu, K., and Shen, Y., and Wang, S.: 3D convolutional neural network for regional precipitation nowcasting. Journal of Image and Signal Processing, 7 (4) 200–212 (2018)
 [17] Zhang, D., Shen, D.: Multimodal multitask learning for joint prediction of multiple regression and classification variables in Alzheimer’s disease. NeuroImage 59, pp. 895–907 (2012)
 [18] Wang, S., Shen, Y., Chen, W., Xiao, T.: Automatic recognition of mild cognitive impairment from mri images using expedited convolutional neural networks. International Conference on Artificial Neural Networks, pp. 373–380 (2017)

[19]
Lei, B., Cheng, N., Frangi, A.F., Tan, E.L., Cao, J., Yang, P., et al.: Selfcalibrated brain network estimation and joint nonconvex multitask learning for identification of early Alzheimer’s disease. Med. Image Anal.
61, 101652 (2020) 
[20]
Lei, B., et al.: Deep and joint learning of longitudinal data for Alzheimer’s disease prediction. Pattern Recognition,
102 107247 (2020)  [21] Wang, S., et al.: Ensemble of 3D Densely Connected Convolutional Network for Diagnosis of Mild Cognitive Impairment and Alzheimer’s Disease. Deep Learning Applications, 53–73 (2020)
 [22] Hu, S., Yuan, J., Wang, S.: Crossmodality Synthesis from MRI to PET Using Adversarial UNet with Different Normalization. 2019 International Conference on Medical Imaging Physics and Engineering (ICMIPE), 1–5 (2019)
 [23] Yu, S., et al.: Multiscale Enhanced Graph Convolutional Network for Early Mild Cognitive Impairment Detection. International Conference on Medical Image Computing and ComputerAssisted Intervention, 228–237 (2020)
 [24] Lee, M.H., Smyser, C.D., Shimony, J.S.: Restingstate fMRI: a review of methods and clinical applications. Am. J. Neuroradiol. 34 (10), pp. 1866–1872 (2013)
 [25] Cao, P., et al.:Generalized fused group lasso regularized multitask feature learning for predicting cognitive outcomes in Alzheimer’s disease. Comput. Meth. Programs Biomed. 162, pp. 19–45 (2018)
 [26] Munsell, B.C., Wu, G., Gao, Y., Desisto, N., Styner, M.: Identifying relationships in functional and structural connectome data using a hypergraph learning method. In: Ourselin S. et al. (eds) MICCAI 2020, LNCS, vol. 9901, pp. 9–17. Springer, Cham (2016). 10.1007/9783319467238_2
 [27] Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)
 [28] Wang, S., Wang, X., Hu, Y., Shen, Y., Yang, Z., Gan, M., Lei, B.: Diabetic retinopathy diagnosis using multichannel generative adversarial network with semisupervision. IEEE Transactions on Automation Science and Engineering, 182 (2), 574–585 (2021)
 [29] Yu, W., et al.: Tensorizing GAN with highorder pooling for Alzheimer’s disease assessment. IEEE Transactions on Neural Networks and Learning Systems, DOI: 10.1109/TNNLS.2021.3063516 (2021)
 [30] Hu, S., Shen, Y., Wang, S., Lei, B.: Brain MR to PET Synthesis via Bidirectional Generative Adversarial Network. International Conference on Medical Image Computing and ComputerAssisted Intervention, 698–707 (2020)
 [31] Hu, S., Yu, W., Chen, Z., Wang, S.: Medical Image Reconstruction Using Generative Adversarial Network for Alzheimer Disease Assessment with ClassImbalance Problem. IEEE 6th International Conference on Computer and Communications (ICCC), 1323–1327 (2020)
 [32] Lei, B., et al.: Skin lesion segmentation via generative adversarial networks with dual discriminators. Medical Image Analysis, 64, 101716 (2020)
 [33] Dong, Y., Sawin, W., Bengio, Y.: HNHN: Hypergraph Networks with Hyperedge Neurons(2020). arXiv preprint https://arxiv.org/abs/2006.12278
 [34] Tzouriomazoyer, N., et al.: Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI singlesubject brain. NeuroImage 15, pp. 273–289 (2002)
 [35] Wang, J., Wang, X., Xia, M., Liao, X., Evans, A., He, Y.: GRETNA: a graph theoretical network analysis toolbox for imaging connectomics. Front. Hum. Neurosci. 9, 386 (2015)
 [36] Cui, Z., Zhong, S., Xu, P., Gong, G., He, Y.: PANDA: a pipeline toolbox for analyzing brain diffusion images. Front. Hum. Neurosci. 7, 42 (2013)
 [37] Yan, C., Zang, Y.: DPARSF: a MATLAB toolbox for ”pipeline” data analysis of restingstate fMRI. Front. Syst. Neurosci. 4, 13 (2010)
 [38] Ashburner, J., et al.: SPM12 Manual. Wellcome Trust Centre for Neuroimaging, London (2014)
 [39] Jiang, J., Wei, Y., Feng, Y., Cao, J., Gao, Y.: Dynamic Hypergraph Neural Networks. In: IJCAI, pp. 2635–2641 (2019)
 [40] Golkov, V., et al.: Qspace deep learning: twelvefold shorter and modelfree diffusion MRI scans. IEEE Trans. Med. Imag. 35 (5), 1344–1351 (2016)

[41]
Suykens, J.A., Vandewalle, J.: Least squares support vector machine classifiers. Neural Process. Lett.
9, pp. 293–300 (1999)  [42] Gray, K.R., Aljabar, P., Heckemann, R.A., Hammers, A., Rueckert, D.: Random forestbased similarity measures for multimodal classification of Alzheimer’s disease. NeuroImage 65, pp. 167–175 (2013)
 [43] Kipf, T.N., Welling, M.: Semisupervised classification with graph convolutional networks (2016). arXiv preprint https://arxiv.org/abs/1609.02907
 [44] Veitch, D.P., et al.: Understanding disease progression and improving alzheimer’s disease clinical trials: Recent highlights from the alzheimer’s disease neuroimaging initiative. Alzheimer’s Dementia 15 (1), pp. 106–152 (2019)
Comments
There are no comments yet.