Establishing the neurobiological mechanisms underlying intelligence is a key area of research in Neuroscience [1, 2]. General intelligence at a young age is predictive of later educational achievement, occupational attainment, and job performance [3, 4, 5, 6]. Moreover, intelligence in childhood or early adulthood is associated with health outcomes later in life as well as mortality [7, 8, 9, 5, 10]. Thus, understanding the mechanisms of cognitive abilities in children potentially has important implications for society and can be used to enhance such abilities, for example through targeted interventions such as education and the management of environmental risk factors [4, 11].
Neuroimaging can play a key role in advancing our understanding of the neurobiological mechanisms of cognitive ability. Several brain-imaging studies have shown that total brain volume is the strongest brain imaging derived predictor of general intelligence [12, 13, 14] (). To a somewhat lesser degree, regional cortical volume and thickness differences in the frontal, temporal, and parietal lobes have also been linked to intelligence [15, 16, 17, 13, 12]. Converging neuroimaging evidence led to the proposal of the parieto-frontal integration theory  whereby a distributed network of brain regions is responsible for the individual variability in cognitive abilities. This theory is also supported by human lesion studies [19, 20].
The ABCD-NP Challenge 2019 asked the question “Can we predict fluid intelligence from T1-weighted MRI?”
We took an exploratory, data-driven approach to answering this question — a hackathon organised by our local research centres: the UCL Centre for Medical Image Computing (CMIC) and Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS). Our centres aim to address key medical challenges facing 21st century society through world-leading research in medical imaging, medical image analysis, and computer-assisted interventions. Our expertise extends from feature extraction/generation through to image-based modelling22, 23, 24], and beyond. The hackathon involved researchers across research groups in our centres, in addition to colleagues from the affiliated Wellcome Centre for Human Neuroimaging and the Department of Clinical and Experimental Epilepsy at UCL. The hackathon took place on an afternoon in February 2019, after which we followed up with regular progress meetings.
In this paper we report our findings for predicting fluid intelligence in 9/10-year-olds from T1-weighted MRI using machine learning regression and deep learning methods (convolutional neural networks — CNNs). Our paper is structured as follows. The next section describes the challenge data and our methods. Section 3 presents our results, which we discuss in section 4 before concluding.
The ABCD-NP Challenge data consists of pre-processed T1-weighted MRI scans and fluid intelligence scores for children aged 9–10 years. The imaging protocol can be found in . Pre-processing included skull-stripping, noise removal, correction for field inhomogeneities [26, 27], and affine alignment to the SRI24 adult brain atlas . SRI24 segmentations and corresponding volumes were also provided.
The cohort was split into training (), validation (), and test () sets. The training and validation sets also include scores of fluid intelligence, which are measured in the ABCD Study using the NIH Toolbox Neurocognition battery . For the challenge, fluid intelligence was residualised to remove linear dependence upon brain volume, data collection site, age at baseline, sex at birth, race/ethnicity, highest parental education, parental income and parental marital status.
2.2 Features derived from the data
We trained the models to predict fluid intelligence both from the provided T1-weighted images (voxel intensity) as well as voxel-wise feature maps generated from these images using a probabilistic segmentation approach. There are many different methods to extract various features from T1-weighted images , such as tissue-type labels obtained from probabilistic segmentations. These segmentations can be constructed in a way to capture not only the relative tissue composition in a voxel, but also information about shape differences between individuals. This requires mapping each subject to a common template — a fundamental technique of computational anatomy . Here we constructed such a template from all available T1-weighted MRI scans (), which generated normalised (non-linearly aligned to a common mean) tissue segmentations for each subject.
We used a generative model  to probabilistically segment each T1-weighted MRI in the challenge data set into three tissue types: grey-matter, white-matter, other — see Figure 1. The in-house model111Available from https://github.com/WCHN/segmentation-model. used here contains key improvements over the one in : (1) we place a smoothing prior on the template; (2) we obtain better initial values by first working on histogram representations of the images; (3) we normalise over population image intensities in a principled way, within the model; (4) we place a prior on the proportions of each tissue, which is also learned during training. There are two types of normalised segmentations: non-modulated and modulated. Modulated segmentations include the relative shape change when aligning to the common template.
Seven features per voxel were considered: T1-weighted intensity plus our six derived features corresponding to modulated and non-modulated probabilities for each tissue type. All images, including the feature maps from probabilistic segmentations, were spatially smoothed with a Gaussian kernel of 12mm FWHM and masked to remove voxels outside of the brain.
2.3 Predicting fluid intelligence: Machine Learning Regression
We explored several machine learning regression algorithms of varying complexity, including Multi-Kernel Learning (MKL) , Kernel Ridge Regression (KRR) , Gaussian Process Regression (GPR) 
and Relevance Vector Machines. The inputs to these models consisted of different concatenated combinations of our seven voxel-wise features described in section 2.2. Analyses were run in PRoNTo version 3 [37, 22]
, a software toolbox of pattern recognition techniques for the analysis of neuroimaging data, as well as custom-written code.
In our preliminary analyses, we trained different combinations of regression algorithms and features using 5-fold cross-validation within the training set to select the best combination of algorithm and features as measured by lowest cross-validated mean-squared error (MSE). We then rtrained the best-performing model using the entire training set and used this trained model to generate predictions of fluid intelligence scores for the validation and test sets. Our best-performing model was KRR using all six voxel-wise derived features (tissue-type probabilities) concatenated into an input feature vector of length
million per individual. We set the regularization hyperparameter to, which was optimised through 5-fold nested cross-validation within the training set in preliminary analyses.
We investigated robustness/stability of our KRR model using modified jackknife resampling (80/20 train/test split). Explicitly, we trained the model on a random subsample of of the training set and generated predictions for both the held out of the training set and the full validation set. We repeated this procedure 1000 times to generate confidence bounds on performance (MSE).
2.4 Predicting fluid intelligence: Convolutional Neural Networks
Separately, we explored the use of CNNs. The motivation was to incorporate spatial information that is not explicitly modelled as features. The CNNs were trained directly on the pre-processed T1-weighted images. Similarly to previous work that predicted brain age, Alzheimer’s disease progression, or brain regions from MRI scans [38, 39, 40] we applied various layers of 3D-convolutional kernels with filter size of 3x3x3 voxels on down-sampled images with dimensions of 61x61x61 voxels. We trained and validated multiple neural networks including those in . Our best performing CNN (lowest MSE) consisted of four convolutional layers and three fully-connected layers followed by dropout layers with a probability of
. The first six layers were activated by rectified linear units. The convolutional layers were followed by batch normalization and max-pooling operations. We used the Adam optimizer with an initial learning rate ofand a decay of , and we stopped training at the epoch. To evaluate the network, we randomly sampled 10 subsets of 1870 subjects from the training set, applied them to train 10 CNN models and evaluated performance on the validation set. We report MSE averaged over the ten passes.
Our best-performing regression model was KRR using our six derived voxel-wise features as input. This produced on the validation set. By comparison, our best-performing CNN achieved (average
) on the validation set. However, we observed that the better-performing CNNs on the training set did not generalise well to the validation set, possibly due to over-fitting, or mismatch between training and validation data (as suggested by the considerable difference in variances).
Table 1 shows MSE and Pearson’s correlation coefficient (mean std) for training and validation predictions from our top-performing methods. We found that KRR performed the best and we used this model to generate our challenge submission, which resulted in on the test set.
|Method||Training set||Validation set||Test set|
|(best: 64.39)||(best: 0.2542)||(best: 70.82)||(best: 0.0157)|
Figure 2 shows model robustness results from modified jackknife resampling (1000 repetitions, 80/20 split). Confidence in our predicted MSE on the validation set is within residual IQ point.
Figure 3 shows the residual fluid intelligence score prediction errors for KRR on the training and validation sets. The V shape of the curves shows that smaller residual scores have lower errors, since the model is predicting close to the mean value.
We found that predicting residual fluid intelligence from structural MRI images is challenging. The correlation between predicted and actual intelligence scores was low for all methods we tested (). This contrasts with previous studies for predicting (non-residualised) fluid intelligence, which have demonstrated that both total brain volume and regional cortical volume/thickness differences are relatively strong predictors () [13, 14, 12].
The lower predictive performance we observed might be influenced by the residualisation, which prevents modelling of covariance between the residualisation factors and the image-based features. Moreover, there is evidence that including variables in the residualisation procedure that are correlated with the regression targets/labels is likely to remove important variability in the data leading to predictive models with low performance .
We note that previous studies used small sample sizes (of order 10–100), whilst the ABCD-NP Challenge dataset comprised a very large-scale dataset (of order 1000–10000). Whereas subject recruitment for small samples tend to be well controlled, resulting in homogeneous sample characteristics, large samples are more heterogeneous by nature, thus predictive models are more challenging to build. Accordingly, a recent study has demonstrated that the accuracy of classification results tends to be smaller for larger sample sizes .
Our image-based features were voxel-wise probabilistic tissue-type labels: grey matter, white matter, and other. Beyond tissue-type labels, there might be value in investigating other features generated by the generative segmentation model. One such interesting feature is scalar momentum , which has been shown to be predictive for a range of different problems .
Finally, we mention a possible limitation of our approach. In recent years, it has become standard practice to create study-specific group templates in neuroimaging , especially in Voxel-Based Morphometry (VBM) analyses. Originally, this approach was proposed for group analysis using mass univariate statistics (e.g., statistical parametric mapping). However, one needs to exercise caution when applying such an approach in machine learning, as it might lead to slightly optimistic predictions by creating dependence across the overall dataset. In order to avoid this potential issue, one would need to create templates based only on the training set. This might be computationally challenging when cross-validation strategies are used. To the best of our knowledge, no studies have investigated whether study-specific templates indeed result in slightly inflated predictions, and it remains an interesting question for future work.
Our paper presents the winning method for the ABCD Neurocognitive Prediction Challenge 2019. We found that kernel ridge regression outperformed more complex models, such as convolutional neural networks, when predicting residual fluid intelligence scores for the challenge dataset using our custom tissue-type features derived from the preprocessed T1-weighted MRI. The correlation between the predicted and actual scores is very low ( for the KRR on the validation set), implying that the association between structural images and residualised fluid intelligence scores is low. It may be that structural images contain very little information on residualised fluid intelligence, but further study is warranted.
-  Goriounova, N.A., Mansvelder, H.D.: Genes, cells and brain areas of intelligence. Frontiers in Human Neuroscience 13, 44 (2019). https://doi.org/10.3389/fnhum.2019.00044
-  Deary, I.J., Penke, L., Johnson, W.: The neuroscience of human intelligence differences. Nature Reviews Neuroscience 11(3), 201–211 (2010). https://doi.org/10.1038/nrn2793
-  McCall, R.B.: Childhood IQ’s as Predictors of Adult Educational and Occupational Status. Science 197(4302), 482–483 (1977). https://doi.org/10.1126/science.197.4302.482
-  Gottfredson, L.S.: Why g matters: The complexity of everyday life. Intelligence 24(1), 79–132 (1997). https://doi.org/10.1016/S0160-2896(97)90014-3
-  Deary, I.J., Strand, S., Smith, P., Fernandes, C.: Intelligence and educational achievement. Intelligence 35(1), 13–21 (2007). https://doi.org/10.1016/j.intell.2006.02.001
-  Johnson, W., McGue, M., Iacono, W.G.: Genetic and environmental influences on academic achievement trajectories during adolescence. Developmental Psychology 42(3), 514–32 (2006). https://doi.org/10.1037/0012-1622.214.171.1244
-  Batty, G.D., Deary, I.J., Gottfredson, L.S.: Premorbid (early life) IQ and Later Mortality Risk: Systematic Review. Annals of Epidemiology 17(4), 278–288 (2007). https://doi.org/10.1016/j.annepidem.2006.07.010
-  Batty, G.D., Wennerstad, K.M., Smith, G.D., Gunnell, D., Deary, I.J., Tynelius, P., Rasmussen, F.: IQ in Early Adulthood and Mortality By Middle Age. Epidemiology 20(1), 100–109 (2008). https://doi.org/10.1097/ede.0b013e31818ba076
-  Lam, N.H., Borduqui, T., Hallak, J., Roque, A.C., Anticevic, A., Krystal, J.H., Wang, X.J., Murray, J.D.: Effects of Altered Excitation-Inhibition Balance on Decision Making in a Cortical Circuit Model. bioRxiv 100347 (2017). https://doi.org/10.1101/100347
-  Deary, I.J., Pattie, A., Starr, J.M.: The Stability of Intelligence From Age 11 to Age 90 Years: The Lothian Birth Cohort of 1921. Psychological Science 24(12), 2361–2368 (2013). https://doi.org/10.1177/0956797613486487
-  Fors, S., Torssander, J., Almquist, Y.B.: Is childhood intelligence associated with coexisting disadvantages in adulthood? Evidence from a Swedish cohort study. Advances in Life Course Research 38, 12–21 (2018). https://doi.org/10.1016/J.ALCR.2018.10.005
-  MacLullich, A.M.J., Ferguson, K.J., Deary, I.J., Seckl, J.R., Starr, J.M., Wardlaw, J.M.: Intracranial capacity and brain volumes are associated with cognition in healthy elderly men. Neurology 59(2), 169–174 (2002). https://doi.org/10.1212/WNL.59.2.169
-  McDaniel, M.A.: Big-brained people are smarter: A meta-analysis of the relationship between in vivo brain volume and intelligence. Intelligence 33(4), 337–346 (2005). https://doi.org/10.1016/j.intell.2004.11.005
-  Rushton, J.P., Ankney, C.D.: Whole brain size and general mental ability: A review. International Journal of Neuroscience 119(5), 692–732 (2009). https://doi.org/10.1080/00207450802325843
-  Andreasen, N.C., Flaum, M., Swayze, V., O’Leary, D.S., Alliger, R., Cohen, G., Ehrhardt, J., Yuh, W.T.: Intelligence and brain structure in normal individuals. The American Journal of Psychiatry 150(1), 130–4 (1993). https://doi.org/10.1176/ajp.150.1.130
-  Narr, K.L., Woods, R.P., Thompson, P.M., Szeszko, P., Robinson, D., Dimtcheva, T., Gurbani, M., Toga, A.W., Bilder, R.M.: Relationships between IQ and Regional Cortical Gray Matter Thickness in Healthy Adults. Cerebral Cortex 17(9), 2163–2171 (2007). https://doi.org/10.1093/cercor/bhl125
-  Karama, S., Colom, R., Johnson, W., Deary, I.J., Haier, R., Waber, D.P., Lepage, C., Ganjavi, H., Jung, R., Evans, A.C.: Cortical thickness correlates of specific cognitive performance accounted for by the general factor of intelligence in healthy children aged 6 to 18. NeuroImage 55(4), 1443–1453 (2011). https://doi.org/10.1016/j.neuroimage.2011.01.016
-  Jung, R.E., Haier, R.J.: The Parieto-Frontal Integration Theory (P-FIT) of intelligence: converging neuroimaging evidence. The Behavioral and Brain Sciences 30(2), 135–154 (2007). https://doi.org/10.1017/S0140525X07001185
Gläscher, J., Tranel, D., Paul, L.K., Rudrauf, D., Rorden, C., Hornaday, A., Grabowski, T., Damasio, H., Adolphs, R.: Lesion mapping of cognitive abilities linked to intelligence. Neuron61(5), 681–91 (2009). https://doi.org/10.1016/j.neuron.2009.01.026
-  Woolgar, A., Parr, A., Cusack, R., Thompson, R., Nimmo-Smith, I., Torralva, T., Roca, M., Antoun, N., Manes, F., Duncan, J.: Fluid intelligence loss linked to restricted regions of damage within frontal and parietal cortex. Proceedings of the National Academy of Sciences 107(33), 14899–14902 (2010). https://doi.org/10.1073/pnas.1007928107
-  Oxtoby, N.P., Alexander, D.C., the EuroPOND consortium: Imaging plus X: multimodal models of neurodegenerative disease. Current Opinion in Neurology 30(4), 371–379 (2017). https://doi.org/10.1097/WCO.0000000000000460
-  Schrouff, J., Rosa, M.J., Rondina, J.M., Marquand, A.F., Chu, C., Ashburner, J., Phillips, C., Richiardi, J., Mourão-Miranda, J.: PRoNTo: pattern recognition for neuroimaging toolbox. Neuroinformatics 11(3), 319–37 (2013). https://doi.org/10.1007/s12021-013-9178-1
-  Schrouff, J., Monteiro, J.M., Portugal, L., Rosa, M.J., Phillips, C., Mourão-Miranda, J.: Embedding Anatomical or Functional Knowledge in Whole-Brain Multiple Kernel Learning Models. Neuroinformatics 16(1), 117–143 (2018). https://doi.org/10.1007/s12021-017-9347-8
-  Blumberg, S.B., Tanno, R., Kokkinos, I., Alexander, D.C.: Deeper image quality transfer: Training low-memory neural networks for 3d images. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. pp. 118–125. Springer International, Cham (2018)
-  Casey, B.J., et al.: The Adolescent Brain Cognitive Development (ABCD) study: Imaging acquisition across 21 sites. Developmental Cognitive Neuroscience 32, 43–54 (2018). https://doi.org/10.1016/j.dcn.2018.03.001
-  Hagler, D.J., et al.: Image processing and analysis methods for the Adolescent Brain Cognitive Development Study. bioRxiv 457739 (2018). https://doi.org/10.1101/457739
-  Pfefferbaum, A., et al.: Altered Brain Developmental Trajectories in Adolescents After Initiating Drinking. American Journal of Psychiatry 175(4), 370–380 (2018). https://doi.org/10.1176/appi.ajp.2017.17040469
-  Rohlfing, T., Zahr, N.M., Sullivan, E.V., Pfefferbaum, A.: The SRI24 multichannel atlas of normal adult human brain structure. Human Brain Mapping 31(5), 798–819 (2010). https://doi.org/10.1002/hbm.20906
-  Akshoomoff, N., Beaumont, J.L., Bauer, P.J., Dikmen, S.S., Gershon, R.C., Mungas, D., Slotkin, J., Tulsky, D., Weintraub, S., Zelazo, P.D., Heaton, R.K.: VIII. NIH Toolbox Cognition Battery (CB): composite scores of crystallized, fluid, and overall cognition. Monographs of the Society for Research in Child Development 78(4), 119–132 (2013). https://doi.org/10.1111/mono.12038
-  Monté-Rubio, G.C., Falcón, C., Pomarol-Clotet, E., Ashburner, J.: A comparison of various MRI feature types for characterizing whole brain anatomical differences using linear pattern recognition methods. NeuroImage 178, 753–768 (2018). https://doi.org/10.1016/j.neuroimage.2018.05.065
-  Ashburner, J.: A fast diffeomorphic image registration algorithm. NeuroImage 38(1), 95–113 (2007). https://doi.org/10.1016/J.NEUROIMAGE.2007.07.007
-  Blaiotta, C., Freund, P., Cardoso, M.J., Ashburner, J.: Generative diffeomorphic modelling of large MRI data sets for probabilistic template construction. NeuroImage 166, 117–134 (2018). https://doi.org/10.1016/j.neuroimage.2017.10.060
-  Rakotomamonjy, A., Bach, F.R., Canu, S., Grandvalet, Y.: SimpleMKL. Journal of Machine Learning Research 9, 2491–2521 (2008), http://www.jmlr.org/papers/v9/rakotomamonjy08a.html
-  Shawe-Taylor, J., Cristianini, N.: Kernel Methods for Pattern Analysis. Cambridge University Press (2004)
-  Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning. The MIT Press (2006), http://www.GaussianProcess.org/gpml
-  Tipping, M.E.: Sparse Bayesian Learning and the Relevance Vector Machine. Journal of Machine Learning Research 1, 211–244 (2001), http://www.jmlr.org/papers/v1/tipping01a.html
-  http://www.mlnl.cs.ucl.ac.uk/pronto/)
-  Sturmfels, P., Rutherford, S., Angstadt, M., Peterson, M., Sripada, C., Wiens, J.: A Domain Guided CNN Architecture for Predicting Age from Structural Brain Images. arXiv e-prints 1808.04362 (2018)
-  Payan, A., Montana, G.: Predicting Alzheimer’s disease: a neuroimaging study with 3D convolutional neural networks. arXiv e-prints arXiv:1502.02506 (2015), preprint
-  Milletari, F., Ahmadi, S.A., Kroll, C., Plate, A., Rozanski, V., Maiostre, J., Levin, J., Dietrich, O., Ertl-Wagner, B., Bötzel, K., Navab, N.: Hough-CNN: Deep Learning for Segmentation of Deep Brain Regions in MRI and Ultrasound. arXiv e-prints arXiv:1601.07014 (2016)
-  Rao, A., Monteiro, J.M., Mourao-Miranda, J., Alzheimer’s Disease Initiative: Predictive modelling using neuroimaging data in the presence of confounds. NeuroImage 150, 23–49 (2017). https://doi.org/10.1016/j.neuroimage.2017.01.066
-  Varoquaux, G.: Cross-validation failure: Small sample sizes lead to large error bars. NeuroImage 180, 68–77 (2018). https://doi.org/10.1016/j.neuroimage.2017.06.061