Log In Sign Up

Degenerative Adversarial NeuroImage Nets for 3D Simulations: Application in Longitudinal MRI

by   Daniele Ravì, et al.

The recent success of deep learning together with the availability of large medical imaging datasets have enabled researchers to improve our understanding of complex chronic medical conditions such as neurodegenerative diseases. The possibility of predicting realistic and accurate images would be a breakthrough for many clinical healthcare applications. However, current image simulators designed to model neurodegenerative disease progression present limitations that preclude their utility in clinical practice. These limitations include personalization of disease progression and the ability to synthesize spatiotemporal images in high resolution. In particular, memory limitations prohibit full 3D image models, necessitating various techniques to discard spatiotemporal information, such as patch-based approaches. In this work, we introduce a novel technique to address this challenge, called Profile Weight Functions (PWF). We demonstrate its effectiveness integrated within our new deep learning framework, showing that it enables the extension to 3D of a recent state-of-the-art 2D approach. To our knowledge, we are the first to implement a personalized disease progression simulator able to predict accurate, personalised, high-resolution, 3D MRI. In particular, we trained a model of ageing and Alzheimer's disease progression using 9652 T1-weighted (longitudinal) MRI from the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset and validated on a separate test set of 1283 MRI (also from ADNI, random partition). We validated our model by analyzing its capability to synthesize MRI that produce accurate volumes of specific brain regions associated with neurodegeneration. Our experiments demonstrate the effectiveness of our solution to provide a 3D simulation that produces accurate and convincing synthetic MRI that emulate ageing and disease progression.


page 1

page 4

page 6

page 7


Degenerative Adversarial NeuroImage Nets: Generating Images that Mimic Disease Progression

Simulating images representative of neurodegenerative diseases is import...

Predicting Knee Osteoarthritis Progression from Structural MRI using Deep Learning

Accurate prediction of knee osteoarthritis (KOA) progression from struct...

Diagnosing Glaucoma Progression with Visual Field Data Using a Spatiotemporal Boundary Detection Method

Diagnosing glaucoma progression is critical for limiting irreversible vi...

Learning Spatio-Temporal Model of Disease Progression with NeuralODEs from Longitudinal Volumetric Data

Robust forecasting of the future anatomical changes inflicted by an ongo...

Iteratively Optimized Patch Label Inference Network for Automatic Pavement Disease Detection

We present a novel deep learning framework named Iteratively Optimized P...

Grey matter sublayer thickness estimation in themouse cerebellum

The cerebellar grey matter morphology is an important feature to study n...

1 Introduction

Predicting and preventing neurodegenerative diseases is one of the global health challenges of the 21st century. Disease progression modelling has been used to describe the time course of neurodegeneration and track the related disease severity, enabled by the availability of large longitudinal clinical and imaging studies. Although the biomarker data contained in these studies are only available at discrete and irregular time points, and from individuals that each cover only a fraction of the entire process, data-driven approaches have shown promising results to infer a comprehensive trajectory of the disease [13]. Some of these approaches attempt to model neurodegeneration directly on full resolution MRI, opening the door for important clinical and technical applications. From the clinical point of view such systems might be useful for: i) early and differential diagnosis; ii) improved precision for clinical trials; and iii) training clinicians. From the technical point of view, such systems might be used for validating other computational models, or to generate synthetic data (data augmentation) for improving the generalizability of novel AI frameworks (avoiding the need for large numbers of expensive scans).

Initial solutions developed for this problem were restricted to learning only one morphological deformation across all subjects or, eventually, a few morphological templates associated to specific sub-groups [3, 9, 15, 12]. More advanced approaches learn subject-specific deformations. One of the first attempts in this direction was proposed in [10], which combined a biophysical model and a deformation field obtained by non-rigid registration of two real images to impose the desired level of atrophy and generate the simulated image. Due to its complex computational model, this approach is extremely resource-demanding and is not scalable to high-resolution images. Exploiting the success of deep learning, Bowles et al. [2] proposed a framework based on Generative Adversarial Networks (GANs). This approach used image arithmetic to add or subtract atrophy patterns by manipulating the MRI directly but was restricted to linear disease progression and morphological changes that are the same across all subjects. To obtain subject-specific morphological changes, Dalca et al. [4]

proposed a probabilistic model that can learn either a universal or a conditional template, jointly with a neural network that provides efficient alignment of the images to these templates. A more advanced deep learning approach was proposed in 


, where adversarial training is used to learn the joint distribution of brain images and ages. Similarly, 

[14] proposed DANI-Net, a framework adapted initially from a face-ageing model. Rather than using a traditional conditional GAN [1, 11], DANI-Net [14]

is based on a conditional adversarial autoencoder that enables conditioning the progression not only upon age but also fixed characteristics like clinical diagnosis, while smoothing the progression to improve temporal consistency. Additionally, whilst 

[19] uses only cross-sectional data to train the system, [14] introduces novel biological constraints (regional and voxel-based) to exploit longitudinal data for improving the precision of neurodegeneration. Although [19, 14] both propose subject-specific progression, they are only available for 2D images.

The challenge of synthesizing accurate and realistic 3D medical images in high resolution is not straightforward. Memory limitations necessitate working only on a localized area of the input (patches, 2D slices, etc.) that discards spatiotemporal information. Applications that require the use of this data can decompose the problem into learning multiple models in a lower dimension of the input. However, unifying these models can be challenging, particularly if their training is hampered by instability that creates spatial/temporal discontinuity between their output. In this work, we introduce PWF developed to deal with this problem. We demonstrate its effectiveness integrated within our new deep learning framework, showing that it enables the extension to 3D of a recent state-of-the-art 2D approach called Degenerative Adversarial NeuroImage Net (DANI-Net). Our 3D simulation framework demonstrates superior accuracy and capability to model subject-specific disease progression on high-resolution 3D MRI.

The rest of the paper is structured as follows. In Section 2 we summarize the baseline method of [14], which was developed to model neurodegeneration for a single 2D-slice of MRI. Then, in Section 3 we present our novel framework, “3D.S-DANI-Net”, which extends the baseline approach to 3D via our novel PWF technique, and adds other improvements. In particular, Section 3.4

describes PWF, which is sufficiently general to be applied in many other high-dimensional computer vision applications involving deep learning. In Section 

4 we describe the data set and our training protocol. Finally, the experimental results are presented in Section 5, and Section 6 concludes the paper.

2 Baseline Method (DANI-Net)

DANI-Net is an extension of a face ageing regression model [20] to include biological and anatomical constraints associated with neurodegenerative disease progression. DANI-Net operates on a single 2D-slice of MRI and consists of the following blocks:

2.1 Pre-processing

The input slice

is normalized to remove irrelevant variation through three steps: i) linear co-registration to 1mm isotropic MNI template; ii) skull-stripping; iii) intensity normalisation to zero mean and unit standard deviation.

2.2 Conditional Deep Autoencoder (CDA)

This block is composed of two deep neural networks: an encoder that embeds in a latent space , and a generator that projects back to the original manifold.

The latent vector

is conditioned on two variables: — a numerical representation [0-3] of diagnosis (i.e. cognitively normal, subjective memory concern, early/late mild cognitive impairment, Alzheimer’s disease); and an index describing age, binned into =10 groups. The CDA learns the morphological brain changes that occur during ageing or disease progression. This employs a deformation loss that minimizes the difference between the input and a weighted average of two outputs from the nearest age bins:


where reflects the distance between the input age and the output group and and are the synthetic images generated by the CDA.

2.3 Adversarial Training

This block involves two discriminator networks, and , trained in an adversarial manner with the CDA.

drives to produce

with a uniform prior to smooth temporal progression. Training uses the following loss function:


where is the expectation operator, is a vector sampled from , estimates the probability that a vector comes from , and is the latent vector obtained from .

drives to produce realistic brain images. Training uses the following loss function:


where estimates the probability that a slice contains realistic brain structures.

2.4 Biological Constraints

This block models neurodegeneration through a set of regional and voxelwise constraints that ensure decreasing image intensity (brain tissue density [17]) with age. For a synthetic output in age group , the first loss function imposes that all with has equal or higher intensity, and that with has equal or lower intensity. Recall that intensity is normalized in the first block. This loss function is a good regularizer for the progression, but it introduces voxelwise rigidity caused by unmodeled intensity changes due to tissue deformation. A second loss function overcomes this limitation and models regional neurodegeneration through a set of pre-trained support-vector regressors (SVRs). and are defined as follows:


where and are the slice dimensions, and sgn is the sign function.


where is the number of regions, is the i-th region-mask, is the corresponding intensity change, and avoids numerical errors.

2.5 DANI-Net Total Loss

The total loss used to train the CDA of DANI-Net is a weighted sum of the constituent loss functions:


where is cross entropy obtained by the discriminator on the generated latent vectors, and is cross entropy obtained by the discriminator on synthetic images . The weights allow for framework customization, such as:

  • increasing increases the contribution of disease progression (the SVRs);

  • increasing regularizes voxel intensity changes for flat regions, but may increase rigidity of brain structures;

  • increasing increases model generalization at the cost to decrease favours qualitatively realistic brain images;

  • increasing reduces temporal smoothing to allow rapid progression, which can introduce temporal discontinuity;

  • increasing increases similarity across age, which diminishes progression learned by the SVRs.

It is clear that some loss functions optimize concurrent tasks and thus finding the optimal configuration for these weights is not straightforward.

Figure 1: 3D.S-DANI-Net pipeline, extending the DANI-Net architecture. The proposed modifications improve the disease progression model (blocks , and ) and ensure 3D consistency across multiple 2D MRI slices (block PWF).

3 Proposed Method (3D.S-DANI-Net)

Figure 1 shows 3D.S-DANI-Net, which modifies the architecture proposed in [14] in two key ways: i) increasing accuracy of the underlying model of neurodegeneration through refined loss functions , , ; and ii) extending from 2D to 3D MRI using the new PWF block that unifies training of multiple 2D models (one per slice). The subsections below describe each modification.

3.1 Deformation Loss

Eq. 1 proposed in [14] minimizes the difference between each input having input age , and a weighted average of the two adjacent age bins. This discretization process allows learning of morphological changes between age groups and prevents the CDA from memorizing (in the latent space) the value of as an individual representation for each sample and thereby overfitting the related age conditioning. The real value was used to compute the degree of membership for the two closest output bins. In 3D.S-DANI-Net, we consider not only the two closest age bins but the entire output sequence with . Explicitly, we impose a degree of similarity between the input and each of the generated outputs . The degree of similarity is determined through a fuzzy Gaussian membership function centred on the average age of each bin and with proportional to the maximum age difference (e.g. ). The reason behind this is that since the generated sequence of images belongs to a single individual, the entire sequence should preserve some similarity with the input. Additionally, the similarity must be higher for output associated with age groups closer to and lower for the others. Consequently, we propose the following modification to :


3.2 Regional Loss

In [14] the rates of intensities change of each brain region were imposed by a set of SVRs (Eq. 5). However, most neurodegeneration models proposed in the literature show that brain atrophy increases monotonically until a plateau is reached [8]. Accordingly, we replace the SVRs with logistic regressors (LRs). One advantage is that the regression will be less sensitive to regions having fewer training samples, e.g., age extremes. Logistic regressors are also much faster to train and to run. We also propose a second refinement to this loss that weights regressor errors by a value related to the region size to prioritise consistent intensity within large regions.

According to these two considerations, is updated as follows:


where LR is the logistic regressor for region .

3.3 Voxel Loss

Eq. 4 proposed in [14] uses the sign function to impose monotonicity of progression in each voxel. Here we redefine with min-max operators as follows:


Although Eq. 9 and Eq. 4 optimize the same properties, the former has the advantage to be much faster during training.

3.4 3D Consistency

DANI-Net requires approximately 11GB of GPU memory to train the neurodegeneration model on a single 2D-slice. With current hardware technology we are not able to train a full spatiotemporal model for a high-resolution 3D-MRI (total memory required is 11GB (#slices) = 150GB). An alternative solution is to train each 2D-slice separately and unify them in a sensible manner. Here we exploit this principle imposing consistency at two different levels:

  • training consistency: we proposed PWF, a strategy for training many different networks so that they follow a similar path on the error surface, thereby ensuring a consistent optimal solution.

  • spatial consistency: we reinforce spatial consistency using a weighted average between models trained on consecutive slices.

3.4.1 Training Consistency (PWF)

Due to the high dimensionality of the problem and the use of a complex loss function (a weighted sum of multiple losses), the training of DANI-Net was often unstable [14]. This is especially problematic for extending to 3D. Convergence failures in some slices will generate spatial-inconsistency in the synthetic MRI. The problems of instability of a deep neural network are often associated to the difficulty in terms of the features of the landscape or error surface that the optimization must navigate to be able to deliver a good solution that avoids local minima. The adversarial components of DANI-Net ( and ) are comparable to two GANs, which makes matters worse since it is well known that they suffer from training instability [5, 7]. Consequently, traditional techniques developed to deal with this type of training issues are not adequate in DANI-Net. Fig. 2 shows an example of a hypothetical training manifold having a few local minima that some networks are not able to avoid.

Figure 2: Traditional training procedure where multiple loss functions have a constant relevance along the entire training process. In a complex system, this may lead to finding a local minimum as an optimal solution.

To guarantee that multiple DANI-Net models associated with different slices converge to a stable and consistent solution, we propose PWF. The intuition behind this technique is as follows: assuming we have multiple sub-loss functions, PWF uses a set of profile functions that weight the relevance of each loss during the training and guide the network to follow a specific path towards the global minimum on the error surface. In other words, these predefined profile functions help the system to avoid exploring irrelevant areas of the manifold and prevent it from getting stuck in a local minimum. Fig.3 depicts how PWF helps to avoid the local minima and, in our case, ensure that different models reach a consistent optimal solution.

The definition of these profile functions is application-specific and can be derived by analyzing how a human would solve that visual problem (i.e. following a multistage learning strategy where the simpler sub-tasks are optimized first and then the more complicated ones).

In our case, to identify the profile functions and their parameters, we perform an experimental evaluation on the validation set based on visual inspection of the results and on verifying the training convergence of different networks. For this application we found that a good strategy to train our framework is as follows: to start, high importance on the weight (associated with the deformation loss) is first imposed, in this way the network learns a simple progression model based on conditional morphological deformations. Then, using an exponential decay function, the relevance of is reduced whereas the contribution of and are increased allowing a further and gradual expansion of the brain regions based on the learnt regressors. The relevance of and associated, respectively, with temporal smoothing and image realism, are also increased. As explained in Section 2.5, increasing reduces temporal smoothing so that neurodegeneration can commence without losing temporal continuity, while increasing the relevance for gradually relaxes the constraint on image realism. Although this last point might seem counterintuitive, it is important for generalization performance since generated images may be quite different from the training samples and, in this case, the discriminator would recognize them as unrealistic although they could potentially be real. This behaviour can drive the generator to avoid creating these brain structures although they are reasonable, which amounts to overfitting of individual morphology.

Figure 3: Proposed PWF strategy. The relevance of each loss during training is specified by a profile function. In this case, the networks follow a specific path in the manifold and avoid local minima.

In our experiments, we also realize that traditional initialization approaches (zero initialization, random initialization, Xavier initialization, etc.) are inadequate when PWF is used in the pipeline. Instead, in this scenario, the training models have to start from a common training point (red cloud in Fig. 3

). To do so, we build an initialization model obtained through 10 training epochs on the training-set images obtained from one of the central slices of the MRI. This pre-trained model is then used as initialization for all the models in 3D.S-DANI-Net.

The aforementioned training strategies are translated into the training process using the following profile functions:

where is the current epoch, describes how fast the profile functions inside PWF will act, are the initial weights for the different losses and finally controls the width expansion of the profile functions.

3.4.2 Spatial Consistency

The last strategy used to ensure the consistency of 3D.S-DANI-Net is to impose spatial correlation between models associated with consecutive 2D slices. This process is described schematically in Fig. 4. For each slice with the current index in the axial view of the input , we use a window of models of 3D.S-DANI-Net named with . A weighted sum of the outputs is used to build the final synthetic slice . All output are then assembled to create the final synthetic . In our final configuration, we use a window =5 and a Gaussian weight function having =1.5.

3.4.3 Region Extraction Based on Atlas

The brain regions in DANI-Net [14]

were determined in a data-driven fashion using a hierarchical clustering approach. However, we realize that this can lead to regional inconsistency when different slices are processed separately. To avoid this problem, in 3D.S-DANI-Net the regions are pre-defined by a brain atlas and imposed on each MRI using linear registration. The extracted regions are then used to train the logistic regressors

, and also to embed in the system the regional ratio of intensity changes learnt by these regressors. This process not only increases the biological meaning of the disease progression model but also avoids regional inconsistencies between consecutive slices.

Figure 4: Pipeline describing the spatial consistency strategy used in 3D.S-DANI-Net.
Figure 5: Synthetic MRI obtained by different configurations of our approach against the baseline and generated starting from 2 participants in the test set. The red circle highlights an artefact created when the network does not converge to a global minimum. Green circles show aliasing artefacts due to slice misalignment, and yellow boxes show artefacts on spatial discontinuity generated again by unstable training.

4 Training Details, Parameters and Evaluation

Data used in the preparation of this article were obtained from the ADNI database ( The ADNI was launched in 2003 as a public-private partnership, led by Principal Investigator Michael W. Weiner, MD. The primary goal of ADNI has been to test whether serial magnetic resonance imaging (MRI), positron emission tomography (PET), other biological markers, and clinical and neuropsychological assessment can be combined to measure the progression of mild cognitive impairment (MCI) and early Alzheimer’s disease (AD).

In our experiments, we selected 12386 pre-processed T1-weighted MRI scans from participants in the ADNI dataset. The scans were obtained using different pre-processing pipelines, scanners, and at multiple sites. Participants were aged between 63 and 87 years old, and 28% were cognitively normal, 4% have been diagnosed with subjective memory concern, 54% with mild cognitive impairment and 14% with Alzheimer’s disease. Each participant has on average 4.7 MRI spanning 3 years. We divided our dataset in train-set (MRI: 9852; participants: 876), test-set (MRI-slices: 1283; participants: 179) and validation set (MRI-slices: 1251; participants: 179). In the test-set, we make sure that participants have at least one follow-up visit two years after baseline, to allow sufficient time for observable neurodegeneration to occur.

The validation set is used to tune the system through the profile functions assessed by visual inspection of the output (see Section 3.4.1).

Once these functions are defined, we train, using 3D.S-DANI-Net, = 95 models, each associated with one of the different slices of the . Each has its own set of region masks and its own set of logistic regressors

but they are all trained using the same profile function and the same training configuration that is based on the stochastic gradient descent solver, ADAM (

= 0.0002, 1 = 0.5). We stop the training procedure after 400 epochs where each iteration uses a random mini-batch with 100 slices (2D) having the size of 128128. In each

and for each test participant, we also apply a final transfer learning step where we align the test MRI-slice to the model by age and diagnosis. This step involves an additional 50 training iterations on a single input image. To note, only the MRI from each test subject’s first visit is used to personalise the trained model. This transfer learning step is essential to tune the model for the specific morphology of the individual’s brain. For a fair comparison, all the considered frameworks include this transfer learning step in the training pipeline.

In our evaluation, is the input MRI from the baseline visits, is the follow-up MRI (ground truth) against which we evaluate the generated .

Finally, since DANI-Net was not designed to handle 3D input, for comparison purposes, we generated the corresponding full 3D MRI by stacking the 2D output obtained from the different models (one per each slice) on top of each other.

Configuration Large regions Large regions
Small regions Textures with high-freq. Textures with low-frequency

[14] 0.078 0.011 0.072 0.003 4.002 3.251 1.248 0.697 1.854 1.868 1.854 1.868 0.583

0.073 0.008 0.071 0.006 3.637 3.048 1.21 0.698 1.59 1.473 1.59 1.473 0.358

- 0.062 0.004 0.057 0.003 1.707 1.398 1.298 0.812 2.748 2.598 2.748 2.598 0.500

- 0.081 0.008 0.074 0.011 4.306 3.282 1.268 0.729 1.937 1.971 1.937 1.971 0.710
-- 0.074 0.005 0.061 0.003 2.395 2.381 1.258 0.702 1.781 1.726 1.781 1.726 0.334

Table 1: Regional volumetric errors (mean, standard deviation) in synthetic vs real MRI, as percentage of total brain volume

5 Results

Here we report the results of our experiments. We investigate each component of our framework by characterizing model performance quantitatively across multiple model configurations. We also provide a complementary qualitative analysis of image realism.

Our quantitative experiments involve volumes of selected brain regions that are relevant to neurodegeneration in both ageing and Alzheimer’s disease. In synthetic and real MRI, we calculate volumes using the FSL library [16] for the following regions: left hippocampus (), right hippocampus (), peripheral grey matter (), ventricular cerebrospinal fluid (), total gray matter (), and total white matter (). We control for individual head size by dividing volumes by total brain volume. The volumetric error between synthetic MRI and real MRI is examined for multiple model configurations and benchmarked against the existing baseline method (DANI-Net [14]).

We evaluate the individual contribution of each of the three components of 3D.S-DANI-Net: i) the new loss functions, denoted ; ii) the spatial consistency constraint ; and iii) the training consistency block incorporating our new profile functions, .

Table 1 reports absolute errors (mean and standard deviation) in volumes, expressed as a percentage of total brain volume. We also report a total error () computed as the average error across regions, each first normalized in the range [0,1] to reflect clinical applications where abnormality in regions are considered equally important [6].

Our results show that, with respect to the baseline approach, reduces error in all regions, although always within one standard deviation of the mean error. This suggests that our new loss functions are working as intended. We note that, although the errors in and appear identical, they are in fact opposite in sign, indicating that volumetric errors are concentrated around the / boundary, which could be due in part to partial volume effects [18]. Additionally, our model improves performance considerably in the region, which is particularly challenging to predict due to the outer cortical brain surface being highly textured and highly variable between individuals.

In our experiments, we found that the proposed consistency strategies ( and ), if used separately, increased volumetric errors by up to of total brain volume. This was most notable in large, flat regions (containing texture features with low-frequency) that span multiple slices. We hypothesize that this may be due to the additional complexity of enforcing constraints across models (of multiple MRI slices), relative to within a model. With reference to , increases the error negligibly for (% of total brain volume), but almost doubles the error in both and (albeit only from to of total brain volume). The situation is better for , where the error increases by % in , and % in both and . We conclude that to generate realistic MRI with an accurate global appearance, the proposed training consistency approach (PWF) is more beneficial than the proposed spatial consistency constraints. Vice versa, if accuracy is desirable in smaller regions of interest (such as and ) or in highly-textured features such as , then the between-model consistency () should be prioritised — as shown in Table 1 for configuration - versus -.

The last row of the table shows that, although the full configuration -- does not provide the best results on the individual regions in terms of absolute volumetric error, the use of the combination of and minimizes our clinically-relevant total error metric . In fact, with respect to the baseline approach, our full configuration reduces by almost half from to . This promotes the use of 3D.S-DANI-Net in clinical applications. We examined this further by assessing the improvements obtained by --

with respect to the baseline method statistically using a paired t-test. For

, and , all p-values were less than 0.0001 whereas for and the improvement was not statistically significant.

For the qualitative analysis, two randomly-selected cases are reported in Fig. 5. As we can see here, the output generated by the -- configuration produces visually superior synthetic MRI having fewer artefacts than synthetic MRI generated by the baseline approach, and by the other configurations. Notable artefacts appear in sagittal and coronal axes, which are most likely due to these approaches lacking any consistency constraints. Additionally, some of the 2D models may fail to converge, thereby creating discontinuity along these axes (yellow boxes in Fig. 5). In the case of training instability, the artefacts may also be visible on the axial plane (red circle in Fig. 5).

The configurations -, - and -- reduce these issues. Images from - have limited discontinuity artefacts thanks to the averaging of neighbouring models, but they lose high-frequency details. Images from - preserves high-frequency features and reduces the discontinuity thanks to PWF, but consecutive slices are not aligned since PWF does not enforce spatial consistency which creates aliasing artefacts (green circles in Fig. 5). Finally, -- produces the fewest artefacts by trading off benefits from PWF and spatial consistency. We also report (not shown) that in age groups having fewer MRI available (i.e. age80 and age70), the predictions from all configurations are slightly more blurred.

More cases of our qualitative analysis are provided in the supplementary material where the improvements of -- are tangible.

Finally, in Fig. Degenerative Adversarial NeuroImage Nets for 3D Simulations: Application in Longitudinal MRI we show an example of the entire simulation obtained using 3D.S-DANI-Net. Neurodegeneration is apparent in the progression, and the main manifestations are ventricular expansion, hippocampus contraction, and cortical thinning.

6 Conclusion and Future Work

We have proposed and evaluated a novel framework for simulating neurodegeneration in ageing and disease on high-resolution 3D MRI. We extended a recent 2D framework called DANI-Net [14]. Our new 3D simulation framework called 3D.S-DANI-Net encodes a spatiotemporal disease progression model in a complex deep neural network, including the use of profile weight functions (PWF) to overcome memory limitations that generally preclude full 3D spatiotemporal modelling. PWF is a deep learning training strategy that promises widespread utility in high-dimensional computer vision applications where the training of a complex adversarial network suffers from instability. Our approach guides model fitting convergence, especially in the case when traditional techniques developed to deal with these problems are inadequate.

Experiments showed improved quantitative and qualitative performance of our new framework over previous work. Our framework produces realistic, high-resolution, 3D, synthetic MRI that accurately predict relevant brain volumes (in real MRI) during normal ageing and Alzheimer’s disease progression. We highlighted the contributions of each component of our framework, which enable customizable modelling choices configured to specific applications.

Although we specifically modelled ageing and disease-related neurodegeneration in MRI, we believe 3D.S-DANI-Net can be used with different medical imaging modalities (e.g. PET, CT, etc.) and to model disease progression in other organs (i.e., lung, prostate, heart, spinal cord, retina). In our experiments, we limited model conditioning to only age and clinical diagnosis, but the framework can handle additional conditional features (e.g. disease phenotype, genotype, demographics, lifestyle measures, cognitive and behavioural score) to personalize the model further.


The authors would like to thank NVIDIA Corporation for the donation of the Titan Xp GPU used for this research.

Data collection and sharing for this project was funded by the ADNI (National Institutes of Health Grant U01 AG024904) and DOD ADNI (Department of Defense award number W81XWH-12-2-0012). ADNI is funded by the National Institute on Aging, the National Institute of Biomedical Imaging and Bioengineering, and through generous contributions from the following: AbbVie, Alzheimer’s Association; Alzheimer’s Drug Discovery Foundation; Araclon Biotech; BioClinica, Inc.; Biogen; Bristol-Myers Squibb Company; CereSpir, Inc.; Cogstate; Eisai Inc.; Elan Pharmaceuticals, Inc.; Eli Lilly and Company; EuroImmun; F. Hoffmann-La Roche Ltd and its affiliated company Genentech, Inc.; Fujirebio; GE Healthcare; IXICO Ltd.; Janssen Alzheimer Immunotherapy Research & Development, LLC.; Johnson & Johnson Pharmaceutical Research & Development LLC.; Lumosity; Lundbeck; Merck & Co., Inc.; Meso Scale Diagnostics, LLC.; NeuroRx Research; Neurotrack Technologies; Novartis Pharmaceuticals Corporation; Pfizer Inc.; Piramal Imaging; Servier; Takeda Pharmaceutical Company; and Transition Therapeutics. The Canadian Institutes of Health Research is providing funds to support ADNI clinical sites in Canada. Private sector contributions are facilitated by the Foundation for the National Institutes of Health ( The grantee organization is the Northern California Institute for Research and Education, and the study is coordinated by the Alzheimer’s Therapeutic Research Institute at the University of Southern California. ADNI data are disseminated by the Laboratory for Neuro Imaging at the University of Southern California.

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 666992.

EPSRC grant EP/M020533/1 supports DCA’s work on this topic. The NIHR UCLH Biomedical Research Centre also supports this work.


  • [1] G. Antipov, M. Baccouche, and J.-L. Dugelay. Face aging with conditional generative adversarial networks. In 2017 IEEE International Conference on Image Processing (ICIP), pages 2089–2093. IEEE, 2017.
  • [2] C. Bowles, R. Gunn, A. Hammers, and D. Rueckert. Modelling the progression of alzheimer’s disease in mri using generative adversarial networks. In Medical Imaging 2018: Image Processing, volume 10574, page 105741K. International Society for Optics and Photonics, 2018.
  • [3] O. Camara, M. Schweiger, R. I. Scahill, W. R. Crum, B. I. Sneller, J. A. Schnabel, G. R. Ridgway, D. M. Cash, D. L. Hill, and N. C. Fox. Phenomenological model of diffuse global and regional atrophy using finite-element methods. IEEE transactions on medical imaging, 25(11):1417–1430, 2006.
  • [4] A. V. Dalca, M. Rakic, J. Guttag, and M. R. Sabuncu. Learning conditional deformable templates with convolutional networks. arXiv preprint arXiv:1908.02738, 2019.
  • [5] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved training of wasserstein gans. In Advances in neural information processing systems, pages 5767–5777, 2017.
  • [6] H. Hampel, K. Bürger, S. J. Teipel, A. L. Bokde, H. Zetterberg, and K. Blennow. Core candidate neurochemical and imaging biomarkers of alzheimer’s disease. Alzheimer’s & Dementia, 4(1):38–48, 2008.
  • [7] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, pages 6626–6637, 2017.
  • [8] C. R. Jack Jr, D. S. Knopman, W. J. Jagust, R. C. Petersen, M. W. Weiner, P. S. Aisen, L. M. Shaw, P. Vemuri, H. J. Wiste, S. D. Weigand, et al. Tracking pathophysiological processes in alzheimer’s disease: an updated hypothetical model of dynamic biomarkers. The Lancet Neurology, 12(2):207–216, 2013.
  • [9] B. Karaçali and C. Davatzikos. Simulation of tissue atrophy using a topology preserving transformation model. IEEE transactions on medical imaging, 25(5):649–652, 2006.
  • [10] B. Khanal, N. Ayache, and X. Pennec. Simulating longitudinal brain mris with known volume changes and realistic variations in image intensity. Frontiers in neuroscience, 11:132, 2017.
  • [11] M. Mirza and S. Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.
  • [12] M. Modat, I. J. Simpson, M. J. Cardoso, D. M. Cash, N. Toussaint, N. C. Fox, and S. Ourselin. Simulating neurodegeneration through longitudinal population analysis of structural and diffusion weighted mri data. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 57–64. Springer, 2014.
  • [13] N. P. Oxtoby and D. C. Alexander. Imaging plus X. Current Opinion in Neurology, 30(4):371–379, 2017.
  • [14] D. Ravi, D. C. Alexander, N. P. Oxtoby, and the Alzheimer’s Disease Neuroimaging Initiative. Degenerative Adversarial NeuroImage Nets: Generating Images that Mimic Disease Progression. In MICCAI, pages 164–172. Springer, 2019.
  • [15] S. Sharma, V. Noblet, F. Rousseau, F. Heitz, L. Rumbach, and J.-P. Armspach. Evaluation of brain atrophy estimation algorithms using simulated ground-truth data. Medical image analysis, 14(3):373–389, 2010.
  • [16] S. M. Smith, M. Jenkinson, M. W. Woolrich, C. F. Beckmann, T. E. Behrens, H. Johansen-Berg, P. R. Bannister, M. De Luca, I. Drobnjak, D. E. Flitney, et al. Advances in functional and structural mr image analysis and implementation as fsl. Neuroimage, 23:S208–S219, 2004.
  • [17] P. Vemuri, H. Wiste, S. Weigand, D. S. Knopman, J. Trojanowski, L. Shaw, M. A. Bernstein, P. Aisen, M. Weiner, R. C. Petersen, et al. Serial mri and csf biomarkers in normal aging, mci, and ad. Neurology, 75(2):143–151, 2010.
  • [18] A. Weibull, H. Gustavsson, S. Mattsson, and J. Svensson. Investigation of spatial resolution, partial volume effects and smoothing in functional mri using artificial 3d time series. NeuroImage, 41(2):346–353, 2008.
  • [19] T. Xia, A. Chartsias, S. A. Tsaftaris, A. D. N. Initiative, et al. Consistent brain ageing synthesis. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 750–758. Springer, 2019.
  • [20] Z. Zhang, Y. Song, and H. Qi. Age progression/regression by conditional adversarial autoencoder. In

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    , pages 5810–5818, 2017.