using convolutional neural networks (CNNs) and has been widely explored in neuroimaging field. Most previous studies in neuroimaging field either study the extracted features from predefined regions of interest (ROIs)  or feed 3D convolutional neural networks directly with the 3D imaging volume. The former approach potentially introduces too much prior into the model and limits the input representation. While the latter approach has the advantage of being agnostic and prior-free, adequate priors from previous neuroimaging studies could be helpful to regularize the input information.
The human cortex is commonly modeled as a 2D manifold sheet-like structure, despite the presence of sulci/gyri folds. Therefore, 2D CNNs can, in principle, be applied on the cortical sheet after flattening onto a 2D plane . However, inevitable distortions in the flattening process affect the data representation. Surface cutting has been proposed to alleviate distortions caused by the intrinsic curvature of the cortical surface, but this again introduces artificial changes to the topology of the surface . Modeling the cortical surface of each hemisphere with a sphere is more accurate and desirable , and spherical coordinate system is the common practice in neuroimaging field, as it can preserve the topological structure of the cortical surface. But 2D CNNs cannot be directly applied on a sphere.
A spherical CNNs framework was recently introduced  and is explored for the first time in this study to analyze the human cortex in a spherical representation. Spherical CNNs were proposed to model spherical data such as molecular modeling, 3D shape [5, 6] and has shown promising performances.
Alzheimer’s disease (AD) is a neurodegenerative disease impacting a large population and is the most common cause of dementia. Accurate diagnosis of AD and mild cognitive impairment (MCI) is of increasing importance. Cortical morphometric measures such as cortical thickness derived from T1-weighted structural MRI have demonstrated to be important biomarkers for the diagnosis of AD, MCI, which are characterized by cortical gray matter atrophy. In this work, we apply a spherical CNNs based framework on the cortical thickness data derived from structural MRI in Alzheimer’s Disease Neuroimaging Initiative (ADNI)111http://adni.loni.usc.edu/ cohort, for the AD versus cognitively normal (CN) classification task, and for MCI conversion prediction within two years.
To the best of our knowledge, this is the first work applying spherical CNNs on human cortex data and demonstrates the potential for diverse studies on discriminative analyses of human cortex neuroimaging data.
2.1 Cortical Modeling
The cortical surfaces were reconstructed using FreeSurfer  and morphed to the spherical representation by minimizing areal and distance distortions. All the individual cortical surfaces were registered to a spherical atlas in the fsaverage space matching cortical folding patterns . At each vertex of the atlas cortical surface, multiple measures including thickness, surface area, volume, curvature, sulc, Jacobian determinant (warping to the atlas) can be derived from FreeSurfer. Sensitivities of different measures vary in different diseases. And any measure can naturally be regarded as the channels in the data representation. In this study, we used cortical thickness as it has been previously demonstrated to be highly sensitive for AD diagnosis [8, 9].
We used a sampling grid with a bandwidth of to sample the cortical surfaces, generating a matrix for each hemisphere. For each point in the sampling grid, we queried the closest vertices in the cortical surface in geodesic distance and used the average measure as the matrix value.
2.2 Spherical CNNs
Spherical CNNs are extensions of regular CNNs formulation on the plane to spherical data, migrating the translational equivariance to rotational equivariance. Hence, specially-designed convolution operations are re-formulated on sphere space S2 and 3D rotation group space SO(3) (SO=‘special orthogonal group’). More theoretical underpinnings can be found in .
Elements in the SO(3) space are represented in the Euler ZYZ data format as:
where denotes rotation around the axis, denotes rotation around the axis, are the rotation angles.
Elements in the S2 space can be similarly represented as:
The network architecture is similar to regular CNNs, with spherical convolutional blocks hierarchically layered. The main parameters include bandwidth , which is similar to the spatial dimension in regular CNNs, and number of channels at each convolution block.
In this work, we use a simple network structure with three convolutional layers interleaved with 3D batch normalization (BN) and rectifier linear unit (ReLU). Illustration of the network structure is shown in Fig.1. The number of channels doubles and the spatial dimensions reduce by two along the depth. Specifically, we denote the S2 convolution with bandwidth and channel as S2Conv(, ), and the SO(3) convolution with bandwidth and channel as SO3Conv(, ). The fully convolutional part of the network is sequenced as: S2Conv(32, 32) - BN - ReLU - SO3Conv(16, 64) - BN - ReLU - SO3Conv(8, 128) - BN - ReLU. The three dimensions of the feature maps at each layer are all .
Then we apply a weighted global average pooling (wGAP) step, consisting of integrating over the spatial dimensions of the convolutional feature maps and correcting for the non-uniformity of the grid in the Y axis.
The two hemispheres of human cortex are considered as two sets of spherical data sharing the same diagnosis label. We therefore share the fully convolutional part of the network between left and right hemispheres. The integrated features from left and right hemispheres are concatenated and fed into the last fully connected layer with softmax activation function for the final disease classification.
We also compared to regular CNNs on the same sampled input, with the same architecture using regular 2D convolutions, replacing 3D BN with 2D BN, and doubling the channel dimensions to ensure approximately same number of parameters. Denoting the convolution operation with channels as Conv(
), the fully convolutional part of the network tested for comparison is: Conv(64) - BN - ReLU - Conv(128) - BN - ReLU - Conv(256) - BN - ReLU. The convolution layers have a stride ofand a kernel size of .
2.3 Activation Maps
Spatial localization of features being used by CNNs can be explored using class activation map , which has been applied in medical imaging field . In this study, we extend the class activation map to spherical CNNs, generating class activation maps on the sphere. The activation maps in spherical CNNs are defined in SO(3) space. According to Equation 1 and 2, we selected the activation maps at to explore the activation map patterns and corrected for the non-uniformity of the grid in the Y axis similar to the practice in wGAP. We performed weighted average of the corrected activation maps using the weights from the fully-connected layer.
3.1 Data and Setup
We used the data from ADNI-1 cohort. We screened subjects per diagnosis group as follows: CN subjects as having stayed cognitively normal during a follow-up period of at least two years, MCI stable (MCI-s) subjects as having stayed MCI during a follow-up period of at least two years, MCI progression (MCI-p) subjects as having converted to AD within two years, and AD patients. Subject information for each diagnosis group can be found in Table 1.
|Age (std)||75.64 (5.25)||74.90 (7.33)||74.69 (6.95)||75.18 (7.50)||75.13 (6.82)|
We used the baseline T1-weighted MRI scans acquired using 1.5 T MRI scanners, pre-processed with the standard Mayo Clinic pipeline222http://adni.loni.usc.edu/methods/mri-analysis/mri-pre-processing/ and post-processed by UCSF using FreeSurfer 4.3 .
We performed two binary classification tasks: AD vs. CN and MCI-p vs. MCI-s. In each classification task, we performed 10-fold cross-validation with the fold split generated from random stratified sampling ensuring similar distribution of diagnosis, age, and gender in each split.
In each experiment, we set out one fold as test set, one fold as validation set, and the rest of the folds as training set. At each fold, the model with the maximum validation accuracy is selected as the optimal model. The probability output of all test sets using the optimal models are aggregated together. We reported the area under curve (AUC) of the receiver operating characteristic (ROC) curve in Fig.2, and also reported the accuracy, sensitivity and specificity.
3.2 AD vs. CN classification
The ROC curve for AD vs. CN classification can be found in Fig. 2 (left). The AUC values for spherical vs. standard CNNs are: 0.915 vs. 0.895. The accuracy (ACC), sensitivity (SEN) and specificity (SPE) values (with 0.5 as threshold) for spherical vs. standard CNNs are: 90.0% vs. 84.6%, 89.9% vs. 84.0%, 90.1% vs. 85.4%. The performance is higher than a previous study also using cortical thickness patterns in ADNI cohort (ACC: 84.5%, SEN: 79.4%, SPE: 88.9%, AUC: 0.905) .
3.3 MCI progression prediction
We further test our model on a more challenging MCI progression prediction task using the same network setting. The ROC curve can be found in Fig. 2 (right). The ROC AUC values for spherical vs. standard CNNs are: 0.707 vs. 0.657. The accuracy, sensitivity and specificity values (with 0.5 as threshold) for spherical vs. standard CNNs are: 71.6% vs. 66.4%, 80.2% vs. 69.9%, 61.4% vs. 62.3%. The performance is higher than a previous study on 2-year MCI progression prediction also using cortical thickness patterns in ADNI cohort (ACC: 66.7%, SEN: 59.0%, SPE: 70.2%, AUC: 0.673) .
3.4 Exploratory Visualization
A population-average AD class activation map of left hemisphere at , generated with the spherical CNNs, is shown in Fig. 3 together with a reference label map from the Desikan-Killiany atlas  sampled in the same way as the thickness measures. The colors and orders of the regions in the reference label map are displayed according to the FreeSurfer color lookup table. We observed two blobs of AD predictive regions: the lower left blob corresponding to regions around medial temporal lobe, and the upper blob corresponding to regions in the vicinity of supramarginal gyrus. Both regions are implicated in AD, according to .
Despite promising results obtained via our application of spherical CNNs to cortical measures, there are several limitations and potential future improvements to be considered: the input omits subcortical structures, such as the hippocampus, which is one of the brain structures affected by AD and a sensitive biomarker for AD diagnosis [8, 15], In future work, the hippocampus can be modeled in the same way as the general 3D structures [5, 6], and incorporated into the classification model. And we can use multi-channel input including other measures such as volume to have multi-faceted characterization of the cortex.
We shared the fully convolutional part between left and right hemispheres, while we can also use two different sets of parameters, which however doubles the number of parameters for the network. Left and right hemispheres could also be registered into the same space and concatenated as two channels. By doing so, the asymmetry in the input information could be embedded and utilized by the CNNs for the diagnosis or prediction tasks.
Since the spherical CNNs formulation is still new to the field, there are still variant architectures to test, such as . A more thorough exploration of parameters (bandwidth, channel), architectures, and properties (fully-convolutional property) is still necessary to fully exploit its potential.
In this study, we demonstrate for the first time that the newly introduced spherical CNNs formulation can be an effective deep learning framework for modeling human cortex and performing AD diagnosis task using MRI-based cortical measures. Our results on the ADNI cohort show state-of-the-art classification performance using structural MRI information only. The spherical CNNs formulation has the potential to be applied to further structural MRI studies, on other neurological diseases, and other modalities such as fMRI and PET, as long as the measures can be projected onto the cortical sphere.
Acknowledgments: Thanks for funding from NIH/NHLBI R01-HL121270. Data used in preparation of this article were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu). As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report. A complete listing of ADNI investigators can be found at: http://adni.loni.usc.edu/wp-content/uploads/how_to_apply/ADNI_Acknowledgement_List.pdf
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,
“Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,”in
IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1026–1034.
-  Sandra Vieira, Walter H. L. Pinaya, and Andrea Mechelli, “Using deep learning to investigate the neuroimaging correlates of psychiatric and neurological disorders: methods and applications,” Neuroscience & Biobehavioral Reviews, vol. 74, pp. 58–75, 2017.
-  Heather Cody Hazlett, Hongbin Gu, Brent C Munsell, Sun Hyung Kim, Martin Styner, Jason J Wolff, Jed T Elison, Meghan R Swanson, Hongtu Zhu, Kelly N Botteron, et al., “Early brain development in infants at high risk for autism spectrum disorder,” Nature, vol. 542, no. 7641, pp. 348, 2017.
-  Bruce Fischl, Martin I Sereno, and Anders M Dale, “Cortical surface-based analysis: II: inflation, flattening, and a surface-based coordinate system,” NeuroImage, vol. 9, no. 2, pp. 195–207, 1999.
-  Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling, “Spherical CNNs,” in International Conference on Learning Representations (ICLR), 2018.
-  Carlos Esteves, Christine Allen-Blanchette, Ameesh Makadia, and Kostas Daniilidis, “Learning SO(3) equivariant representations with spherical CNNs,” in The European Conference on Computer Vision (ECCV), September 2018.
-  Anders M Dale, Bruce Fischl, and Martin I Sereno, “Cortical surface-based analysis: I. segmentation and surface reconstruction,” NeuroImage, vol. 9, no. 2, pp. 179–194, 1999.
-  Rémi Cuingnet, Emilie Gerardin, Jérôme Tessieras, Guillaume Auzias, Stéphane Lehéricy, Marie-Odile Habert, Marie Chupin, Habib Benali, et al., “Automatic classification of patients with Alzheimer’s disease from structural MRI: a comparison of ten methods using the ADNI database,” NeuroImage, vol. 56, no. 2, pp. 766–781, 2011.
-  Simon F Eskildsen, Pierrick Coupé, Daniel García-Lorenzo, Vladimir Fonov, Jens C Pruessner, D Louis Collins, Alzheimer’s Disease Neuroimaging Initiative, et al., “Prediction of Alzheimer’s disease in subjects with mild cognitive impairment from the ADNI cohort using patterns of cortical thinning,” NeuroImage, vol. 65, pp. 511–521, 2013.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba,
“Learning deep features for discriminative localization,”in
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
-  Xinyang Feng, Jie Yang, Andrew F. Laine, and Elsa D. Angelini, “Discriminative localization in CNNs for weakly-supervised segmentation of pulmonary nodules,” 2017, Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 568–576.
-  Clifford R Jack, Matt A Bernstein, Nick C Fox, et al., “The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods,” Journal Magn Reson Imaging, vol. 27, no. 4, pp. 685–691, 2008.
-  Rahul S Desikan, Florent Ségonne, Bruce Fischl, Brian T Quinn, Bradford C Dickerson, Deborah Blacker, Randy L Buckner, Anders M Dale, R Paul Maguire, et al., “An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest,” NeuroImage, vol. 31, no. 3, pp. 968–980, 2006.
-  Rahul S Desikan, Howard J Cabral, Christopher P Hess, William P Dillon, Christine M Glastonbury, Michael W Weiner, Nicholas J Schmansky, Douglas N Greve, David H Salat, et al., “Automated MRI measures identify individuals with mild cognitive impairment and Alzheimer’s disease,” Brain, vol. 132, no. 8, pp. 2048–2057, 2009.
-  Xinyang Feng, Jie Yang, Andrew F Laine, and Elsa D Angelini, “Alzheimer’s disease diagnosis based on anatomically stratified texture analysis of the hippocampus in structural MRI,” in International Symposium on Biomedical Imaging (ISBI). IEEE, 2018, pp. 1546–1549.