Graph Convolutions on Spectral Embeddings: Learning of Cortical Surface Data

03/27/2018 ∙ by Karthik Gopinath, et al. ∙ 0

Neuronal cell bodies mostly reside in the cerebral cortex. The study of this thin and highly convoluted surface is essential for understanding how the brain works. The analysis of surface data is, however, challenging due to the high variability of the cortical geometry. This paper presents a novel approach for learning and exploiting surface data directly across surface domains. Current approaches rely on geometrical simplifications, such as spherical inflations, a popular but costly process. For instance, the widely used FreeSurfer takes about 3 hours to parcellate brain surfaces on a standard machine. Direct learning of surface data via graph convolutions would provide a new family of fast algorithms for processing brain surfaces. However, the current limitation of existing state-of-the-art approaches is their inability to compare surface data across different surface domains. Surface bases are indeed incompatible between brain geometries. This paper leverages recent advances in spectral graph matching to transfer surface data across aligned spectral domains. This novel approach enables a direct learning of surface data across compatible surface bases. It exploits spectral filters over intrinsic representations of surface neighborhoods. We illustrate the benefits of this approach with an application to brain parcellation. We validate the algorithm over 101 manually labeled brain surfaces. The results show a significant improvement in labeling accuracy over recent Euclidean approaches, while gaining a drastic speed improvement over conventional methods.



There are no comments yet.


page 3

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Neuroimage analysis consists of studying functional and anatomical information over the brain geometry. The thin outer layer of the brain cerebrum is of particular interest due to its key role in cognition, vision and perception. Statistical frameworks on surfaces are, therefore, highly sought for studying various aspects of the brain. For instance, variations in surface data can reveal new biomarkers as well as potential relations with disease processes [1]. The challenge consists of learning surface data over highly complex and convoluted surfaces. Conventional approaches rely on geometrical simplifications, such as spherical inflation and slow mesh deformations [2], a popular but costly process. For instance, the widely used FreeSurfer [3]

takes around 3 hours to parcellate brain surfaces by slowly deforming brain models towards labeled atlases. Fundamentally, current statistical frameworks exploit spatial information mostly derived from the Euclidean domain, for instance, based on vector fields, image or volumetric coordinates

[4, 5]

. Such information is highly variable across brain geometries and severally hinders the training of modern machine learning algorithms.

State-of-the-art learning approaches [6]

have the potential to offer a drastic speed advantage over traditional surface-based methods, but operate on image or volumetric spaces. Geometric deep learning

[7, 8, 9]

recently proposes to use convolutional filters on irregular graphs. This approach formulates the convolution theorem from Fourier space to spectral domains over graphs. Furthermore, Chebyshev polynomials avoids the explicit computation of graph Laplacian eigenvectors

[10], and local graph filtering is made possible within small neighborhoods [8]. The main concern of these methods is their inability to compare surface data across different surface domains [11, 12, 13, 14]. Laplacian eigenbases are indeed incompatible across multiple geometries. One approach is to map local graph information onto geodesic patches and use conventional spatial convolution via template matching [15, 16]. Geodesic patches are obtained with local parametric constructions of patches [8]. However, fundamentally, spatial representations of surface data remain defined in Euclidean spaces, for instance, using polar representations of pixels or mesh vertices.

This paper leverages recent advances in spectral graph matching to transfer surface data across aligned spectral domains [17]. The transfer of spectral coordinates across domains provides a robust formulation for spectral methods that naturally handles differences across Laplacian eigenvectors, including sign flips, ordering and mixing of eigenvectors in higher frequencies. This spectral alignment strategy was exploited to learn surface data [18], but was limited to pointwise information, ignoring local patterns within surface neighborhoods.

Existing attempts of graph convolutions in neuroimaging [19, 20] exploit single graph and rely on Euclidean representations of brain surfaces. The learning of cortical data across multiple surfaces remain hindered by the incompatibility of spectral bases across geometries. Our approach consists of leveraging spectral coordinates within convolutional networks. This bridges a gap between learning algorithms and geometrical representations. To the best of our knowledge, this is the first attempt of learning surface data via spectral graph convolutions in neuroimaging. This novel approach enables a direct learning of surface data across compatible surface bases by exploiting spectral filters over intrinsic representations of surface neighborhoods. We illustrate the learning capabilities of this approach with an application to brain parcellation. The validation over 101 manually labeled brain surfaces [21] shows a significant improvement of spectral graph convolutions over Euclidean approaches, from a Dice score of 59% to 82%. This performance is aligned with the well established FreeSurfer algorithm [3], which scores 83% [21], while gaining a drastic speed improvement, in the order of seconds. Our contributions are multifold. The transfer of spectral bases across domains enables the design of spectral filters in graph convolutional approaches. Our adaptive spectral filters can consequently learn cortical surface data across multiple geometries, as well as exploit local patterns of data within surface neighborhoods. The next section details the fundamentals of our spectral approach, followed by experiments evaluating the impact of our spectral strategy over standard Euclidean approaches for graph convolutions.

Figure 1: Overview of the algorithm – On the left are inputs: Sulcal depth , and corresponding spectral coordinates . In the middle are: Learned convolution layers,

, where sample filter responses are shown with coarse to fine geometric features. On the right are: Predicted parcel probabilities (Pred) with ground truth (GT) for two parcels. Brain surfaces are inflated for visualization.

2 Method

An overview of the proposed method is shown in Fig. 1

. In a first step, cortical surfaces modeled as a brain graph are embedded in a spectral manifold using the graph Laplacian operator. The spectral embedding of different brain surfaces are then aligned in the manifold using the Iterative Closest Point (ICP) algorithm. Finally, a geometric convolutional neural network (CNN) is used to map input features, corresponding to the spectral coordinates and sulcal depth of brain graph nodes, to a labeled graph.

2.1 Spectral embedding of brain graphs

Let be a brain graph defined with node set , such that , and edge set . Each node has a feature vector representing its 3D coordinates and sulcal depth. We map to a low-dimension manifold using the normalized graph Laplacian operator , where is the weighted adjacency matrix and the diagonal degree matrix. In this work, we define the weight between two adjacent nodes as the inverse of their Euclidean distance. Let be the eigendecomposition of , the normalized spectral coordinates of nodes are given by .

Since the spectral embedding of is only defined up to an orthogonal transformation, we need to align spectral representations of different brain graphs to an arbitrary reference. Denote as the normalized spectral embedding of this reference, we align an embedding to with an iterative method based on the ICP algorithm. In this method, each node is mapped to its nearest reference node in the embedding space. The optimal orthogonal transform between matched nodes is then obtained by solving a Procrustes analysis problem. This process is repeated until convergence.

2.2 Graph convolution on surfaces

We start by presenting the standard CNN model for rigid grids and then explain how this model can be extended to an arbitrary geometry. Let be the input feature map at convolution layer of the network, such that is the -th feature of the -th input node. The network input thus corresponds to . Assuming a 1D grid, the output feature map of layer is given by with


Here, are the convolution kernel weights, the bias weights of the layer, and

is a non-linear activation function, for instance the sigmoid or rectified linear unit (ReLU) functions.

To extend this fixed-grid formulation to a graph , we denote as the neighbors of node . A generalized convolution operation can then be defined as


where is a symmetric kernel in the embedding space with parameter . In this work, we follow [8] and use a Gaussian kernel:


Using this formulation, we define a fully-convolutional network composed of 3 graph convolution layers with feature map sizes of , and , each one having Gaussian kernels. The size of the last layer corresponds to the number of parcels to be segmented (32 in our case). Leaky ReLU is applied after each layer to obtain filter responses: . Since the parcels to segment are mutually exclusive, we use a softmax operation after the last graph convolution layer to obtain the parcel probabilities of each node. The softmax function is given by

. Finally, cross-entropy is employed as output loss function:


where are the trainable network parameters, is the output probability for node and parcel label , and

is a one-hot encoding of the reference segmentation. This loss is minimized by back-propagating the error using standard gradient descent optimization.

3 Results

We now evaluate the performance of our contributions. First, we highlight the advantage of moving graph learning frameworks from a conventional Euclidean domain to a Spectral domain. Second, we assess the improvement of exploiting neighborhoods of surface data versus learning pointwise information on spectral embeddings. Our validation is performed on Mindboggle [21], the largest publicly available dataset of manually labeled brain MRI. It consists of 101 subjects collected from different sites, with cortical meshes varying from 102K to 185K vertices. Each brain surface contains 32 manually labeled parcels. In our experiments, the dataset was randomly split into training, validation and testing with a ratio of 70-10-20

. We further compared our results with the established FreeSurfer software. Our implementation is in Matlab, and uses an i7 desktop computer with 16GB of RAM and an Nvidia Titan X GPU. The computation of spectral embedding and basis alignment takes roughly 15 seconds. Each training epoch takes

5-6 minutes. However, a single forward pass during testing for single subject cortical parcellation takes approximately 3 seconds.

3.1 Euclidean versus Spectral domain

Figure 2: Cortical Parcellation – (First/Left) Learning with Euclidean coordinates, resulting in low Dice score () and inconsistent boundaries (Hausdorff distance of 18.9mm). (Second) Learning with Spectral coordinates, improving Dice score () and boundary regularity (12.3mm). (Third) Spectral domain regularized with MRF, leading to consistent boundaries (10.6mm) with respect to the reference (Fourth). The brain surface is inflated for visualization.

This experiment evaluates the improvement of moving the learning operations from the Euclidean domain to a Spectral domain. To do so, we compare the classification accuracy on 32 cortical parcels when running our algorithm, respectively, in the Euclidean and Spectral domains. Quantitative results are measured in terms of average Dice overlap and Hausdorff distances [18]. Qualitative results are shown in Fig. 2.

Euclidean domain – In our baseline, similarly to the latest approaches of graph convolutions networks [8], we learn from input features in the Euclidean domain. Each cortical point is represented using sulcal depth and its spatial location. The architecture remains the same as described earlier (Fig. 1). We train our network on the randomly split training set until the performance decreases on the validation set. The average Dice overlap across all parcels in our dataset is 59.0% ( 20.5, min/max = 0.0/85.5%). The Hausdorff distance averaged across all parcels is 18.8mm. Fig. 2 clearly illustrates the current limitation of existing graph convolutions approaches. Spatial ambiguities are inherent in the Euclidean domain, i.e., neighboring coordinates in space may not necessarily be neighbors on surfaces. This ambiguity may confuse training on highly convoluted surfaces such as the brain, and possibly explains the strong spatial irregularities observed in the parcels boundaries.

Spectral domain – Our contribution is to operate in a geometry-aware Spectral domain. We now learn on input features represented in the Spectral domain, where each cortical point consists of the sulcal depth with the first three spectral coordinates. We use the same architecture and data split as before. The average Dice overlap across all parcels improves to 82.2%( 5.3, min/max = 69.5/95.1%). Details for each parcels are shown in Fig. 3. The Hausdorff distance averaged across all parcels is now reduced to 12.3mm. This is a 40% improvement over learning in the conventional Euclidean domain. The qualitative results of Fig. 2 show that our cortical parcellation is almost similar to the manual parcellation. The boundary, however, is irregular and requires further regularization.

As an illustration of further refinement, we use Markov random field (MRF) regularization for both Euclidean and Spectral outputs. Toward this goal, we apply a standard graphcut algorithm [22] with minus-log parcel probabilities as unary potentials and the Potts model for defining binary potentials. MRF regularization further improves the overall classification accuracy from 68.2% to 71.1% in the Euclidean domain, and from 84.4% to 85.4% in the Spectral domain. Similar improvement is observed in terms of Hausdorff distance, with a reduction from 18.8mm to 15.9mm in the Euclidean domain, and from 12.3mm to 10.6mm in the Spectral domain. The Dice overlap scores remain, however, similar after regularization, perhaps due to a correct initial overlap of parcels by our algorithm.

3.2 Pointwise versus neighborhood information

Figure 3: Performance Evaluation – Dice scores for all 32 cortical parcels across the dataset when learning with: (Blue) Euclidean, (Orange) pointwise spectral, and (Green/Ours) neighborhoods of spectral coordinates. (Black

) FreeSurfer is shown for comparison. Standard deviations of Dice scores for all parcels: (Euclidean) 20.5%, (pointwise spectral) 14.3%, (FreeSurfer) 10.3%, (spectral neighborhoods) 5.3%, which is the least variable.

In this experiment, we illustrate the benefits of exploiting neighborhoods of surface data over pointwise information in a spectral domain. To follow standard approaches of learning pointwise surface data [18]

, we remove neighboring connections by setting our weighted adjacency matrix as an identity matrix. In addition, we disable the update of kernel parameters in convolutional layers. Training on pointwise information results in an average Dice overlap of 75.6%(

14.3%, min/max = 16.2/92.3%). As a reminder, training on neighborhoods results in 82.2% of Dice score. Exploiting neighborhoods of surface data thus improves the performance of our algorithm. A closer look to the performance scores for each parcel (Fig. 3) also reveals a general improvement when exploiting neighborhoods over pointwise surface data. Results for all parcels are also at par with FreeSurfer’s performance.

In the MindBoggle dataset [21], the manual parcellations were created by experts who corrected parcels initially generated from FreeSurfer. The agreement between manually corrected parcels and FreeSurfer is 83.2% () of Dice score. Our algorithm produces a 82.2% () Dice score with respect to manual parcellation, while gaining a significant improvement in computation time, from 3 hours to 18 seconds for processing one subject. Our graph convolutional framework also results in a stable performance when compared to an Euclidean approach or to FreeSurfer, who respectively have a standard deviation on Dice scores of 20.5% and 10.3% across all parcels. Our spectral approach varies by 5.3%.

4 Conclusion

This paper presented a novel framework for learning surface data via spectral graph convolutions. The algorithm leverages recent advances in spectral matching to enable the comparison of surface data across different surface domains. Our experiments illustrated the benefits of our approach with an application to cortical surface parcellation. This is a particularly challenging problem where current graph convolution approaches remain limited by the inability to compare surface data across brain geometries. This typically results in spatial irregularities of parcel boundaries as illustrated in Fig. 2. By capturing the geometry of the spectral manifold, the proposed method can improve the parcellation accuracy to a Dice score of , from with graph convolutions in the Euclidean space. The performance of our method is comparable to state-of-the-art approaches for cortical parcellation, however, it reduces the computation time by an order of magnitude (18 seconds vs hours for FreeSurfer). While the potential of our method was demonstrated on cortical parcellation, it can be applied to other analyses of surface data. For instance, our framework has direct impact in regression problems that involve predictions of cortical thickness over time, potentially leading to new families of geometry-based biomarkers for neurological disorders.


  • [1] Arbabshirani, M.R., Plis, S., Sui, J., Calhoun, V.D.: Single subject prediction of brain disorders in neuroimaging: Promises and pitfalls. NeuroImage (2017)
  • [2] Tustison, N.J., et al.: Large-scale evaluation of ANTs and FreeSurfer cortical thickness measurements. NeuroImage (2014)
  • [3] Fischl, B., et al.: Automatically parcellating the human cortex. C. Cortex (2004)
  • [4] Zhang, T., Davatzikos, C.: ODVBA: optimally-discriminative voxel-based analysis. IEEE TMI (2011)
  • [5] Hua, X., et al.:

    Unbiased tensor-based morphometry: improved robustness and sample size estimates for AD clinical trials.

    NeuroImage (2013)
  • [6] Kamnitsas, K., et al.: Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Medical image analysis (2017)
  • [7] Bronstein, M., Bruna, J., LeCun, Y., Szlam, A., Vandergheynst, P.: Geometric deep learning: Going beyond euclidean data. IEEE Signal Processing (2017)
  • [8] Monti, F., Boscaini, D., Masci, J., Rodolà, E., Svoboda, J., Bronstein, M.M.: Geometric deep learning on graphs using mixture model CNNs. In: CVPR. (2017)
  • [9] Levie, R., Monti, F., Bresson, X., Bronstein, M.: CayleyNets: Graph convolutional neural networks with complex rational spectral filters. arxiv (2017)
  • [10] Defferrard, M., Bresson, X., Vandergheynst, P.: Convolutional neural networks on graphs with fast localized spectral filtering. In: NIPS. (2016)
  • [11] Bronstein, M., Glashoff, K., Loring, T.: Making laplacians commute. CoRR (2013)
  • [12] Kovnatsky, A., Bronstein, M.M., Bronstein, A.M., Glashoff, K., Kimmel, R.: Coupled quasi-harmonic bases. In: Computer Graphics Forum. (2013)
  • [13] Ovsjanikov, M., Ben-Chen, M., Solomon, J., Butscher, A., Guibas, L.: Functional maps: A flexible representation of maps between shapes. In: SIGGRAPH. (2012)
  • [14] Eynard, D., et al.: Multimodal manifold analysis by simultaneous diagonalization of Laplacians. IEEE PAMI (2015)
  • [15] Masci, J., Boscaini, D., Bronstein, M., Vandergheynst, P.: Geodesic convolutional neural networks on Riemannian manifolds. In: ICCV-3dRR. (2015)
  • [16] Boscaini, D., Masci, J., Rodolà, E., Bronstein, M.: Learning shape correspondence with anisotropic convolutional neural networks. In: NIPS. (2016)
  • [17] Lombaert, H., Arcaro, M., Ayache, N.: Brain transfer: Spectral analysis of cortical surfaces and functional maps. In: IPMI. (2015)
  • [18] Lombaert, H., Criminisi, A., Ayache, N.: Spectral forests: Learning of surface data, application to cortical parcellation. In: MICCAI. (2015)
  • [19] Parisot, S., et al.: Spectral graph convolutions for population-based disease prediction. In: MICCAI. (2017)
  • [20] Ktena, S.I., et al.: Distance metric learning using graph convolutional networks: Application to functional brain networks. In: MICCAI. (2017)
  • [21] Klein, A., et al.: Mindboggling morphometry of human brains. PLOS Bio (2017)
  • [22] Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max- flow algorithms for energy minimization in vision. IEEE PAMI (2004)