DeepAI
Log In Sign Up

Generative Modelling of the Ageing Heart with Cross-Sectional Imaging and Clinical Data

08/28/2022
by   Mengyun Qiao, et al.
10

Cardiovascular disease, the leading cause of death globally, is an age-related disease. Understanding the morphological and functional changes of the heart during ageing is a key scientific question, the answer to which will help us define important risk factors of cardiovascular disease and monitor disease progression. In this work, we propose a novel conditional generative model to describe the changes of 3D anatomy of the heart during ageing. The proposed model is flexible and allows integration of multiple clinical factors (e.g. age, gender) into the generating process. We train the model on a large-scale cross-sectional dataset of cardiac anatomies and evaluate on both cross-sectional and longitudinal datasets. The model demonstrates excellent performance in predicting the longitudinal evolution of the ageing heart and modelling its data distribution.

READ FULL TEXT VIEW PDF
05/23/2019

A model of brain morphological changes related to aging and Alzheimer's disease from cross-sectional assessments

In this study we propose a deformation-based framework to jointly model ...
07/21/2020

A radiomics approach to analyze cardiac alterations in hypertension

Hypertension is a medical condition that is well-established as a risk f...
12/04/2019

Learning to synthesise the ageing brain without longitudinal data

Brain ageing is a continuous process that is affected by many factors in...
09/11/2018

Iterative Segmentation from Limited Training Data: Applications to Congenital Heart Disease

We propose a new iterative segmentation model which can be accurately le...
05/10/2018

TADPOLE Challenge: Prediction of Longitudinal Evolution in Alzheimer's Disease

The Alzheimer's Disease Prediction Of Longitudinal Evolution (TADPOLE) C...
11/24/2018

Conditional Recurrent Flow: Conditional Generation of Longitudinal Samples with Applications to Neuroimaging

Generative models using neural network have opened a door to large-scale...
08/19/2018

Deep Mask For X-ray Based Heart Disease Classification

We build a deep learning model to detect and classify heart disease usin...

1 Introduction

Heart ageing is a predominant risk factor of cardiovascular diseases. Understanding how ageing affects the shape and function of the heart is a key scientific question that has received substantial attention [4, 5, 6]. Due to the high dimensionality of the 3D cardiac shape data, researchers and clinicians often describe the anatomical shape using global metrics such as the volumes or ejection fraction. However, these metrics cannot reflect detailed information of local shape variations. Describing the high-dimensional spatio-temporal anatomy of the heart and its evolution during ageing is still a challenging problem.

In this work, we propose a novel conditional generative model for the ageing heart, which describes the variations of its 3D cardiac anatomy, as well as its associations with age. The model is trained on a large-scale cross-sectional dataset with both cardiac anatomies and non-imaging clinical information. Once trained, given a cardiac anatomy and a target age, the model can perform counterfactual inference and predict the anatomical appearance of the heart at the target age. By evaluating on both cross-sectional and longitudinal datasets, we demonstrate that the predicted anatomies are highly realistic and consistent with real data distribution. The model has the potential to be applied to downstream tasks for cardiac imaging research, such as for analysing of the ageing impact on the anatomy, synthesising shapes for biomechanical modelling and performing data augmentation.

1.1 Related Work

Numerous efforts have been devoted into conditional generative modelling and synthesis of ageing. In this work, we focus on heart ageing synthesis using conditional generative modelling techniques. Existing literature can be broadly classified into the following two categories:

Generative modelling

The field of generative modelling has made tremendous progress recently, driven by deep learning methods such as variational autoencoders (VAEs)

[10, 16]

, generative adversarial networks (GANs)

[12], cycle-consistent GAN (CycleGAN) [23]. Generative models have been widely used in medical imaging. For example, Wang et al. proposed a CycleGAN-based model for cross-domain image generation, which generates pseudo-CT for PET-MR attenuation correction [17]. Yurt et al. proposed a multi-stream GAN architecture for multi-contrast MRI synthesis [21]

. Pawlowski et al. formulated a structural causal model with deep learning components for synthesising and counterfactual inference of MNIST and brain MR images

[15].

Synthesis of ageing

Most ageing synthesis works focused on images of human face while some works explored the synthesis of brain MR images. These works investigated different ways of incorporating age information into the generating process. One way is to concatenate age vector with image feature vector to learn a joint distribution of age and image appearance in face ageing

[2, 22] or brain ageing [20]. Another way is to use a pre-trained age regression network, which provides guidance in age-related latent code generation [1, 19]

. Some works introduced an age estimation loss accounting for age distribution

[7, 13]. In [11], high-order interactions between the given identity and target age were explored to learn personalized age features. Although these methods are not designed for cardiac imaging, they provide valuable insights for modelling the ageing heart.

1.2 Contributions

The contributions of this work are three-fold: 1) We investigate the challenging problem of heart ageing synthesis, where both the structural variation and functional variation (anatomies in different time frames) need to be modelled. To this end, we develop a novel model which consists of an anatomy encoder and a condition mapping network that disentangles age and spatial-temporal shape information in the generating process; 2) We utilise multi-modal information including both imaging data and non-imaging clinical data so that the generative model can account for the impact of multiple clinical factors on the ageing heart; 3) We train the generative model using a large-scale cross-sectional datasets and demonstrate its performance quantitatively on both cross-sectional and longitudinal datasets. To the best of our knowledge, this is the first work to investigate generative modelling for ageing heart synthesis.

2 Methods

2.1 Problem Formulation

Fig. 1 illustrates the proposed generative model. At the inference stage (Fig. 1 right), given a source cardiac anatomy image with its clinical information (source age: , gender: ) and the target age , the network synthesises the cardiac anatomy conditioned on the target age . Thus, the distribution of the synthetic anatomy approximates the distribution of the real data at the target age while maintaining subject-specific structure in the ageing process. Our model utilises a heart anatomy generator , which consists of an encoder , an anatomy decoder and a condition mapping network . During the training stage, the generator learns the evolution of the anatomy from the source age to the anatomy at target age and vice versa in a cyclic manner.

Figure 1: The proposed generative model for the ageing heart. Training: The training scheme includes two parts: cycle-consistent reconstruction, which takes source image at source age as input, synthesise at target age and then back to at age ; self-reconstruction, which takes source image as input and reconstruct at the same age. Inference: The Generator takes input image at source age , together with target age and gender , and generates the image at target age . Please refer to the text for detail.

2.2 Conditional Generative Modelling

2.2.1 Clinical condition incorporation

We incorporate two major clinical conditions, age and gender, into the generative model. The age space is represented using a categorical vector, where denotes the number of age groups. For age group , an age vector is generated:

(1)

where

is a one-hot encoded vector that contains one at the

th-element and zeros elsewhere and is random noise sampled from a prior distribution. The age vector is concatenated with a one-hot gender vector to form the clinical condition .

Condition mapping network

Inspired by [8], we construct a condition mapping network using a multi-layer perception (MLP). It embeds the input clinical condition including age and gender to latent vector in the conditional latent space. This latent representation integrates different clinical factors and enables exploration across the condition space.

Age predictor

We construct an age predictor to help the generative model focus on the age information in learning. takes a cardiac anatomy as input and predicts its age. It is applied only at the training stage for both cycle-consistent reconstruction and self-reconstruction as shown in Fig. 1. We impose a distance loss between the predicted age and target age , as well as between and to guide the age prediction. The age predictor is implemented as a six-layer 3D convolutional network, followed by a fully connected layer to produce an age vector.

2.2.2 Anatomy generator

The input to the generator includes the source anatomy and the condition code generated from the condition mapping network . The generator follows an encoder-decoder structure, where the encoder maps the input into a subject-specific latent code , the decoder concatenates with the clinical condition code and generates an output anatomy . The generation is described by,

(2)

We assume that the encoder preserves the high-level subject-specific feature of the input anatomy and the decoder utilises this information as well as the clinical information to generate the anatomy . Adopting a cyclic design [23], we also generate an image that maps the generated the image back to the source age . This cycle-consistent generation is only applied in training process.

2.2.3 Discriminators

Two discriminator networks are imposed on the latent code and the generated anatomy. is designed to discriminate the latent code by training

to be uniformly distributed. Simultaneously,

will be trained to compute to fool . Such an adversarial process forces the distribution of to gradually approach the prior, which is the uniform distribution. Another discriminator forces the generator to generate realistic cardiac anatomies.

2.3 Training Scheme

An overview of the training scheme is shown in Fig. 1

. To generate realistic anatomies while modelling smooth continuous ageing, we use a multi-task loss function which combines cyclic reconstruction losses

and , adversarial losses for the anatomy discriminator , an adversarial loss for the latent code discriminator and an age loss for the age predictor . For heart ageing synthesis, the source anatomy and the generated anatomy at the same age are expected to be similar. A self-reconstruction loss between source image and reconstruction image is applied to learn the identity generation. In addition, we employ the cycle consistency loss [23] between and for a consistent reconstruction from age to age and back to age . L1 loss is used for reconstruction:

(3)

The generated images , are enforced to the target age space by minimizing the distance between the age predictor outputs , and the age vectors , . A cross-entropy (CE) age loss is defined as,

(4)

An adversarial loss is used to impose an uniform distribution on the latent code :

(5)

where denote random samples from a uniformed prior distribution.

In addition, two adversarial losses conditioned on the source and target ages of the real and synthetic anatomies are introduced, respectively:

(6)
(7)

The adversarial losses presented in Eq. 6 and Eq. 7, minimizing the distance between the input and output images, forces the output anatomies to be close to the real ones.

Overall, the optimisation is formulated as an adversarial training process,

(8)

where the

’s are tunable hyperparameters weighting the loss terms.

3 Experiments

3.1 Datasets

Cross-sectional dataset Short-axis cardiac images at the end-diastolic (ED) and end-systolic (ES) frames of 12,600 subjects from 44.6 to 82.3 years old, were obtained from the UK Biobank and split into training () and test () sets. The age is represented as seven categories () with interval of five years: 44–50, 50-55, 55-60, 60-65, 65-70, 70-75 and 75-83. Most of datasets are from healthy volunteers and about 5-6 have the cardiovascular diseases (CVD), which we would take into consideration in the future.

Longitudinal dataset A longitudinal dataset of 639 subjects from the UK Biobank is used, in which each subject undergoes imaging at two time points. The age ranges from 46.6 to 79.8 years old at the first imaging and 51.3 to 81.9 years at the re-imaging, with a median time gap of 3.2 years. The image resolution and size are the same as the cross-sectional dataset. All evaluations are performed on ED and ES frames of cardiac sequences.

Preprocessing For both datasets, 3D cardiac anatomies at ED and ES frames are extracted from cardiac MR images using a publicly available segmentation network [3]

, then upsampled using a publicly available super-resolution model

[18] followed by manual quality control. Subsequently, affine registration is performed to align all cardiac anatomies to the same orientation. The 3D cardiac anatomies are of an isotropic resolution of mm and of size voxels.

3.2 Experimental Setup

Implementation details

The encoder consisted of five 3D convolutional layers and one flatten layer, outputting the latent code . The decoder consisted of one flatten layer and five 3D transposed convolution layer. The transposed convolution in encoder and decoder used a kernel size of

. All intermediate layers of each block use the ReLU activation function. The dimension of the latent variables

and are both 32. The anatomy segmentations are transferred into one-hot map, and the output of () is restricted to using the hyperbolic tangent activation function. For optimisation, the Adam optimizer [9] is used with learning rate of and weigh decay of . We set in Eq. 1, , , , and in Eq. 8

. The model was implemented using PyTorch

[14]. At the inference stage, only the generator is active, containing , and as described in Eq. 2, while the other parts are not used. Our code will be made publicly available.

Baseline methods

Two ageing synthesis methods, CAAE [22] and a modified version of Lifespan [13], are used as baselines. Since the original codes were developed for 2D face image synthesis, we re-implemented all the codes for 3D cardiac data synthesis. For Lifespan, we replaced the modulated convolution layers with 3D convolution layers to save GPU memory for 3D data.

3.3 Experiments and Results

3.3.1 Heart ageing synthesis

For each subject in the UK Biobank test set, we synthesise a series of anatomies for the same heart at age groups from 40 to 80 with interval of 5 years old. For example, in Fig. 2, the cardiac anatomy of a 50-55 years old female is taken as input and the anatomies of this heart at other ages are predicted using the proposed generative model. From the generated anatomies, we also derive clinical measures, including the left ventricular myocardial mass (LVM), LV end-diastolic volume (LVEDV), LV end-systolic volume (LVESV), RV end-diastolic volume (RVEDV) and RV end-systolic volume (RVESV). The bottom of Fig. 2 illustrates the trends of these clinical measures during heart ageing synthesis. Consistent with the literature [6], we observe a decreasing trend for LV or RV volumes. It demonstrates our model captures the relation between cardiac anatomical structure and age.

Figure 2: An example of the synthetic ageing heart. The first and third rows show the cardiac anatomies at ED and ES frames, in which the blue rectangle denotes the original input anatomy of a 50-55 year old female and the other columns denote the synthetic anatomies at different ages. The second and fourth rows show the difference maps between an aged heart and the baseline anatomy at 45-50 year old. The fifth shows the predicted evolution of clinical measures including LVM, LVEDV, LVESV, RVEDV and RVESV during ageing.

3.3.2 Distribution similarity

Based the synthetic anatomies, we calculate the probability distribution

of each clinical measure against age

and compare it to the probability distribution of the real data

. Here, denotes one of the five clinical measures, e.g. LVM. The distance between and

is evaluated in terms of the Kullback–Leibler divergence (KL) and Wasserstein distance (WD). Table 

1 compares the distribution similarities using different generative models. It shows the proposed method achieves a higher distribution similarity, compared to state-of-the-art ageing synthesis models.

LVM LVEDV LVESV RVEDV RVESV
KL WD KL WD KL WD KL WD KL WD
CAAE [22] 0.0266 19.8875 0.0355 15.6661 0.0737 9.8066 0.0343 18.7905 0.0467 11.9521
Lifespan [13] 0.0253 18.4733 0.0349 15.8506 0.0703 9.8091 0.0322 20.0034 0.0442 12.1956
Proposed 0.0248 15.2829 0.0334 15.7215 0.0675 9.5658 0.0318 19.1561 0.0428 10.8607
Table 1: The distribution similarity between synthetic and real data. The smaller the KL or WD distance, the higher the similarity.

3.3.3 Longitudinal prediction

Using the repeated imaging scans from UK Biobank longitudinal dataset, we evaluate the predictive performance of the model. Given the anatomy at the first time point, the anatomy at the second time point is predicted and compared to the ground truth in terms of Dice metric, Hausdorff distance (HD) and average symmetric surface distance (ASSD), reported in Table 2. It shows that the proposed method achieves a good performance in prediction comparable to or better than other competing methods.

End-diastolic anatomy End-systolic anatomy
Dice HD ASSD Dice HD ASSD
CAAE [22] 0.727(0.057) 30.431(6.058) 2.777(0.788) 0.769(0.070) 15.904(6.766) 2.528(1.054)
Lifespan [13] 0.757(0.064) 29.935(5.988) 2.784(0.806) 0.774(0.072) 16.023(6.527) 2.490(1.041)
Proposed 0.761(0.066) 27.281(7.436) 2.695(0.835) 0.775(0.073) 14.789(7.224) 2.524(1.073)
Table 2: The prediction performance on the UK Biobank longitudinal dataset. The higher Dice or lower HD and ASSD, the better the prediction.

4 Conclusion

To conclude, we propose a novel generative model for the ageing heart anatomy that allows preserving the identity of the heart while changing its characteristics across different age groups. The quantitative results on both cross-sectional and longitudinal datasets demonstrate the method achieves highly realistic synthesis and longitudinal prediction of cardiac anatomies, which are consistent with real data distributions.

4.0.1 Acknowledgements

This work was supported by EPSRC SmartHeart Grant (EP/P001009/1) and DeepGeM Grant (EP/W01842X/1). The research was conducted using the UK Biobank Resource under Application Number 18545. We wish to thank all UK Biobank participants and staff.

References

  • [1] Alaluf, Y., Patashnik, O., Cohen-Or, D.: Only a matter of style: Age transformation using a style-based regression model. ACM Transactions on Graphics 40(4), 1–12 (2021)
  • [2] Antipov, G., Baccouche, M., Dugelay, J.L.: Face aging with conditional generative adversarial networks. In: IEEE International Conference on Image Processing. pp. 2089–2093 (2017)
  • [3] Bai, W., Sinclair, M., Tarroni, G., Oktay, O., et al.: Automated cardiovascular magnetic resonance image analysis with fully convolutional networks. Journal of Cardiovascular Magnetic Resonance 20(1),  65 (2018)
  • [4] Bai, W., Suzuki, H., Huang, J., Francis, C., Wang, S., et al.: A population-based phenome-wide association study of cardiac and aortic structure and function. Nature Medicine 26(10), 1654–1662 (2020)
  • [5] Boon, R.A., Iekushi, K., Lechner, S., Seeger, T., Fischer, A., et al.: MicroRNA-34a regulates cardiac ageing and function. Nature 495(7439), 107–110 (2013)
  • [6] Eng, J., McClelland, R.L., Gomes, A.S., Hundley, W.G., Cheng, S., et al.: Adverse left ventricular remodeling and age assessed with cardiac MR imaging: the multi-ethnic study of atherosclerosis. Radiology 278(3), 714–722 (2016)
  • [7] Huang, Z., Chen, S., Zhang, J., Shan, H.: PFA-GAN: Progressive face aging with generative adversarial network. IEEE Transactions on Information Forensics and Security 16, 2031–2045 (2020)
  • [8] Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., Aila, T.: Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems 33, 12104–12114 (2020)
  • [9] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: International Conference for Learning Representations (2015)
  • [10] Kingma, D.P., Welling, M.: Auto-encoding variational bayes. In: International Conference for Learning Representations (2013)
  • [11]

    Makhmudkhujaev, F., Hong, S., Park, I.K.: Re-Aging GAN: Toward personalized face age transformation. In: International Conference on Computer Vision. pp. 3908–3917 (2021)

  • [12] Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)
  • [13] Or-El, R., Sengupta, S., Fried, O., Shechtman, E., Kemelmacher-Shlizerman, I.: Lifespan age transformation synthesis. In: European Conference on Computer Vision. pp. 739–755 (2020)
  • [14] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., et al.: Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems 32, 8026–8037 (2019)
  • [15] Pawlowski, N., Coelho de Castro, D., Glocker, B.: Deep structural causal models for tractable counterfactual inference. Advances in Neural Information Processing Systems 33, 857–869 (2020)
  • [16] Sohn, K., Lee, H., Yan, X.: Learning structured output representation using deep conditional generative models. Advances in Neural Information Processing Systems 28, 3483–3491 (2015)
  • [17] Wang, C., Yang, G., Papanastasiou, G., Tsaftaris, S.A., Newby, D.E., et al.: DiCyc: GAN-based deformation invariant cross-domain information fusion for medical image synthesis. Information Fusion 67, 147–160 (2021)
  • [18] Wang, S., Qin, C., Savioli, N., Chen, C., O’Regan, D.P., et al.: Joint motion correction and super resolution for cardiac segmentation via latent optimisation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 14–24 (2021)
  • [19]

    Wang, Z., Tang, X., Luo, W., Gao, S.: Face aging with identity-preserved conditional generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 7939–7947 (2018)

  • [20] Xia, T., Chartsias, A., Tsaftaris, S.A., Initiative, A.D.N., et al.: Consistent brain ageing synthesis. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 750–758 (2019)
  • [21] Yurt, M., Dar, S.U., Erdem, A., Erdem, E., Oguz, K.K., Çukur, T.: mustGAN: Multi-stream generative adversarial networks for MR image synthesis. Medical Image Analysis 70, 101944 (2021)
  • [22] Zhang, Z., Song, Y., Qi, H.: Age progression/regression by conditional adversarial autoencoder. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5810–5818 (2017)
  • [23]

    Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: IEEE International Conference on Computer Vision (2017)

References