It has been widely accepted that the optimization of medical imaging systems should be guided by objective measures of image quality (IQ) that quantify the performance of an observer at specific tasks[barrett2013foundations, zhou2018learning, zhou2019learningIO, zhou2019learningHO, zhou2019approximating]. The performance of a numerical observer ideally should account for all sources of randomness in the measured data, which includes the variability within a group of objects being imaged[barrett2013foundations, kupinski2003experimental, dolly2018learning, zhou2018learning, zhou2019learningIO, zhou2019learningHO, zhou2019approximating]. To achieve this, one can establish a stochastic object model (SOM) that can sample numerous realizations of objects from the ensemble of objects to-be-imaged. To establish a SOM that describes the to-be-imaged distribution, it is desirable to establish it from experimental measurements. Moreover, the established SOM should be independent of the imaging system. Kupinski et al. developed a method to fit SOMs by use of noisy imaging measurements[kupinski2003experimental]. However, the reported applications have been limited to lumpy and clustered lumpy object models[kupinski2003experimental]. In order to establish more complicated SOMs, Zhou et al.
implemented a deep learning method using an augmented generative adversarial network (GAN) architecture called an AmbientGAN[bora2018ambientgan] to learn SOMs from noisy and indirect measurements[zhou2019learning]. In that proof-of-concept study, a simple lumpy object model was considered[zhou2019learning]. However, it is well-known that training of GANs can be unstable and, therefore, in practice, it can be difficult to learn the statistical properties of the to-be-imaged distribution.
A recently developed GAN training methodology—Progressive Growing of GANs (ProGANs)—holds great promise to stabilize the training of GANs and has been successfully employed to generate high-quality synthetic images with high resolutions[karras2017progressive]. Unlike the conventional training of GANs, in which all scales of the image distribution are learned simultaneously, the training of ProGANs starts with low-resolution images. Subsequently, network layers are added progressively to the generator and discriminator in synchrony to increase the image resolution for learning finer scale details of the image distribution. Here, the generator and the discriminator have mirrored architectures. It has been shown that GAN training stability and the quality of the produced synthetic images can be improved significantly by use of this training strategy[karras2017progressive]. However, because medical imaging systems record noisy and indirect measurements of object properties, ProGANs cannot be immediately applied to establish stochastic models of objects to-be-imaged.
In this study, inspired by the ProGAN[karras2017progressive] and the AmbientGAN[bora2018ambientgan], we propose a novel training methodology—Progressive Growing of AmbientGANs (ProAGANs)—to stably train AmbientGANs to establish SOMs from noisy and indirect imaging measurements. An idealized magnetic resonance (MR) imaging system and clinical MR brain images from NYU fastMRI Initiative database[zbontar2018fastmri] were considered. The proposed methodology was evaluated by comparing signal detection performance computed by use of ProAGAN-generated synthetic images and images that depict the true object properties.
Consider a linear discrete-to-discrete imaging system:
denotes a vector of an object image having dimension of, denotes an imaging operator, denotes a vector of imaging measurement data, and denotes measurement noise.
2.1 GANs and AmbientGANs
Generative adversarial neural networks (GANs) were developed to learn data distributions by training a generator through an adversarial process with a discriminator[goodfellow2014generative]. Here, the generator is represented by a deep neural network having a set of weight parameters and a mapping function : that maps a latent vector to a synthetic image . The discriminator is represented by another deep neural network having a set of weight parameters and a mapping function : that maps a real or synthetic image to a real-valued scalar that can be used to distinguish between real and synthetic images. However, because medical imaging systems acquire noisy and indirect imaging measurements , GANs cannot be directly employed to establish SOMs.
An augmented GAN architecture named AmbientGAN was developed to learn SOMs from noisy and indirect measurements of object properties[bora2018ambientgan, zhou2019learning]. Similar to GANs, AmbientGANs comprise a generator and a discriminator. The generator is trained to produce synthetic digital images that depict the object properties. Synthetic imaging measurements are subsequently simulated by applying the corresponding imaging operator to the finite-dimensional representations of the object property produced by the generator. The discriminator that maps is applied to distinguish between real and synthetic imaging measurements. The training process of AmbientGANs can be represented by a two-player minimax game:
Here, , where , and represents an objective function. It has been shown that when uniquely induces , and both the discriminator and the generator possess sufficient capacity, approximates when the global optimal of the minimax game is achieved[goodfellow2014generative, bora2018ambientgan].
2.2 Progressive Growing of GANs
Progressive Growing of GANs (ProGANs) were developed to stabilize the training of GANs[karras2017progressive]. Unlike the conventional GANs, in which the generator and the discriminator are trained to learn all scales of image distribution by use of full resolution images through the whole training process, the training of ProGANs start with training the first few layers of the generator and discriminator on low-resolution images that are down-scaled from the original images. Higher-resolution images are subsequently employed and more layers of the generator and discriminator are progressively included in the training to learn finer scale details of the image distribution. It has been shown that the ProGAN training strategy results in significantly improved training stability and produces synthetic images with the state-of-the-art image quality[karras2017progressive]. However, like other GANs, ProGANs cannot be immediately applied to learn SOMs from noisy and/or indirect imaging measurements.
3 Progressive Growing of Ambient-GANs
In this work, we propose a novel training strategy named Progressive Growing of AmbientGANs (ProAGANs) to stably train the AmbientGANs for establishing SOMs from noisy and indirect imaging measurements. A MR imaging system that fully samples k-space data was considered: , where
denotes a 2D discrete Fourier transform (DFT). The generator in the proposed ProAGAN was trained to synthesize images that depict object properties in the to-be-imaged distribution. However, because the MR imaging system records noisy k-space measurement data, the discriminator cannot be applied directly to distinguish between real and synthetic object images. In our proposed architecture of ProAGAN, a 2D inverse discrete Fourier transform (IDFT)is included in the training of ProAGAN to reconstruct object images from k-space measurements, and the discriminator is trained to distinguish between real and synthetic reconstructed images. The goal is to learn the distribution of images that depict to-be-imaged object properties by progressively training a generator by competing against a discriminator that distinguishes between real and synthetic reconstructed images. This training process is illustrated in Fig. 1.
4 Numerical studies
A simulation study was conducted. MR brain images in the NYU fastMRI Initiative database [zbontar2018fastmri] (https://fastmri.med.nyu.edu/) were employed to form an ensemble of images that depict object properties sampled from the unknown SOM. Specifically, 3000 T1 weighted brain MR images corresponding to the magnetic field strength of 3T were selected, and these 3000 images were resized to the dimension of to be employed as real object images. Fully-sampled MR k-space data of these 3000 object images were simulated, and complex Gaussian noise were added to the k-space data. These 3000 noisy k-space measurement data formed the training dataset. An example of MR brain images, its corresponding k-space measurement data and the reconstructed image (i.e., IDFT of k-space measurement data) are shown in Fig. 2.
4.1 Training details
The proposed ProAGAN was implemented in Tensorflow[abadi2016tensorflow] by modifying the ProGAN codes to apply the architecture illustrated in Fig. 1. The codes for the ProGAN can be found at https://github.com/tkarras/progressive_growing_of_gans. The ProAGAN was trained by use of 4 NVIDIA Tesla V100 GPUs. The Adam algorithm [kingma2014adam], which is a stochastic gradient algorithm, was employed to train ProAGANs.
4.2 Hotelling observer validation studies
Signal-known-exactly (SKE) binary signal detection tasks were conducted to assess the learned SOM in a task-dependent way. The considered signal detection tasks require an observer to classify a MR brain image as satisfying signal-absent hypothesis () or signal-present hypothesis ():
where denotes the signal to be detected, and is independent and identically distributed Gaussian noise. Two different signals represented by two artificial tumors were considered. These signals are shown in Fig. 3.
The signal detection tasks were performed on a region of interest (ROI) that was centrally located in the images. The ROIs had dimension of . The covariance matrices corresponding to the real and ProAGAN-produced images were computed by use of 3000 ROIs extracted from the real and ProAGAN-produced images, respectively. These covariance matrices were subsequently employed to computed the Hotelling observer (HO) by use of a covariance matrix decomposition[barrett2013foundations]. The performances of the HOs were evaluated on a testing dataset that comprised 500 pairs of signal-absent and signal-present images. The receiver operating characteristic (ROC) curves and AUC values corresponding to the ProAGAN-produced images were compared to those corresponding to the real images. The ‘proper’ binormal model[pesce2007reliable] was employed to fit the ROC curves.
Examples of real and synthetic images of object properties produced by the ProAGAN are shown in Fig. 4. The synthetic images are promising in terms of visually mimicking the real images.
To illustrate the progressive growing of ProAGAN, synthetic images at different training steps with different resolution levels are shown in Fig. 5. It is demonstrated that the generator was progressively established in a stable way.
The Hotelling templates that were computed by use of the real images and the ProAGAN-produced images are compared in Fig. 6. The ROC curves and AUC values corresponding the real images and the ProAGAN-produced images are compared in Fig. 6 (g), in which the solid curves and the dashed curves correspond to the HOs computed by use of real images and synthetic images produced by the ProAGAN, respectively. The ROC curves are almost identical.
This study provides a novel training methodology—Progressive Growing of AmbientGANs—to stably train AmbientGANs for learning SOMs from noisy and indirect measurement data. In this preliminary study, an idealized magnetic resonance (MR) imaging system was considered, and a task-based validation study was conducted by use of the Hotelling observer. It is demonstrated that the proposed method is able to establish a SOM by use of noisy k-space measurement data. From the perspective of the HO, for the signal detection tasks considered, it was found that the images synthesized by the learned SOM contained nearly the same task-specific information as the true object property images.
This research was supported in part by NIH awards EB020604, EB023045, NS102213, EB028652, and NSF award DMS1614305.