Generative Adversarial Networks Synthesize Realistic OCT Images of the Retina

02/18/2019
by   Stephen G. Odaibo, et al.
0

We report, to our knowledge, the first end-to-end application of Generative Adversarial Networks (GANs) towards the synthesis of Optical Coherence Tomography (OCT) images of the retina. Generative models have gained recent attention for the increasingly realistic images they can synthesize, given a sampling of a data type. In this paper, we apply GANs to a sampling distribution of OCTs of the retina. We observe the synthesis of realistic OCT images depicting recognizable pathology such as macular holes, choroidal neovascular membranes, myopic degeneration, cystoid macular edema, and central serous retinopathy amongst others. This represents the first such report of its kind. Potential applications of this new technology include for surgical simulation, for treatment planning, for disease prognostication, and for accelerating the development of new drugs and surgical procedures to treat retinal disease.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 3

page 4

02/12/2018

Synthesizing Audio with Generative Adversarial Networks

While Generative Adversarial Networks (GANs) have seen wide success at t...
03/31/2020

Learning Generative Models of Tissue Organization with Supervised GANs

A key step in understanding the spatial organization of cells and tissue...
07/16/2019

On the "steerability" of generative adversarial networks

An open secret in contemporary machine learning is that many models work...
12/12/2017

Conditional Generative Adversarial Networks for Emoji Synthesis with Word Embedding Manipulation

Emojis have become a very popular part of daily digital communication. T...
02/11/2019

Synthesizing New Retinal Symptom Images by Multiple Generative Models

Age-Related Macular Degeneration (AMD) is an asymptomatic retinal diseas...
06/10/2018

Improving Surgical Training Phantoms by Hyperrealism: Deep Unpaired Image-to-Image Translation from Real Surgeries

Current `dry lab' surgical phantom simulators are a valuable tool for su...
07/17/2020

Attention2AngioGAN: Synthesizing Fluorescein Angiography from Retinal Fundus Images using Generative Adversarial Networks

Fluorescein Angiography (FA) is a technique that employs the designated ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Optical Coherence Tomography (OCT)[14]

has revolutionized the field of retina. And usage of this technology has become standard in the management of the vast majority of retinal patient encounters. An understanding of the native probability distribution of OCT representation of retinal disease is therefore essential. This is especially so as we seek to acquire more personalized understanding of disease, so as to develop treatments that are most fitting to the pathology of interest.

Following the work of Goodfellow et al[9], Generative Adversarial Networks (GANs), an adversarial type of generative model, has gained popularity based on the increasingly realistic datasets it can generate. Some theory of adversarial training algorithms has been in existence for at least 3 decades (Schmidhuber, 1992)[28]. However, practical feasibility, awareness, traction, and performance are only now notably rising. Interesting applications have included ”deep fakes” of human faces, and more recently, of medical data. For instance, Burlina et al [4] recently used GAN architecture to synthesize fundus images of age-related macular degeneration.

To our knowledge, this is the first use of generative adversarial networks to perform end-to-end synthesis OCT images of the retina. GANs have recently been used for the segmentation of various anatomical parts of the eye within ophthalmic images [31, 19], as well as conditional adversarial generation from vessel trees [6]. They have also been used to generate fundus photographs of the retina [7].

The capability to sample never before seen or even not yet existing retinal pathology will increase both our understanding of disease and facilitate development and assessment of new therapies. One such pathological feature which our GAN architecture detected is macular edema as well as subretinal fluid. Firstly, the presence of macular edema or subretinal fluid in the macula is essentially always indicative of pathology. It is a canonical hallmark that underlies several common retinal conditions such as diabetic macular edema[12, 20, 13, 23, 25], exudative macular degeneration [11, 14, 30, 5, 16], retinal vein occlusions [27], pseudophakic macular edema [32, 29, 17, 18, 24] central serous chorioretinopathy [10, 15, 2, 21, 8], and macula-off retinal detachments [34, 1, 3, 33, 22]. And therefore having a fuller understanding of all the ways and forms of macular edema will be essential to advancing our ability to develop effective therapies for a great number of retinal diseases.

In the remainder of this paper, we present our methods, results, and a discussion of the implications of this development for retinal research.

2 Methods

Figure 1:

AI Generated images. Sampled from Epoch 96000 to 99800

Figure 2: Random Deconvolution output. Epoch 0
Figure 3: Generator and Discriminator losses: All epochs
Figure 4: Generator and Discriminator losses between Epoch 0 and 20
Figure 5: Generator and Discriminator losses between Epoch 0 and 200
Figure 6: Generator and Discriminator losses between Epoch 0 and 2000
Figure 7: Generator and Discriminator losses between Epoch 2000 and 3000
Figure 8: Generator and Discriminator losses between Epoch 0 and 15000

A Generative Adversarial Network was implemented in Keras. The algorithm was run on an 8 core NVIDIA V100 machine equivalent with 600 GB of RAM. A database consisting of 500,000 images after augmentation was utilized for the training. The database had a diverse representation of pathology including macular holes, cystoid macular edema, exudative and non-exudative macular degeneration, central serous retinopathy, macula-off retinal detachments, and normal retina. Training was done for 100,000 epochs. A randomized vector was deconvolved according to the architecture depicted in Figure 1, yielding the generator output. The deconvolutional neural network weights were iteratively trained via backpropagation as directed by its loss function, a binary cross entropy. Our GAN architecture had some similarity as well as some notable differences compared to that recommended heuristically by Radford et al 

[26]

. LeakyReLu was used for the activation function of all layers of both the generator and discrimator, except for their final layer where a tanh and a sigmoid was used respectively. Batchnormalization was done on every layer of the generator and on no layer of the discriminator.

3 Results

Figure 1 shows realistic OCT images of the retina which were synthesized after 99800 training epochs of the algorithm. At the onset, prior to training, the initial random vector of size 100 yielded Figure 2 from the generator. Figure 7 to Figure 8 depict the loss landscape of the discriminator and generator during training.

At the onset of training, where the discriminator is first getting trained, we see a decrease in the discriminator’s loss and a concurrent rise in the generator’s loss. This early behavior is even marked by an early crossing in the loss graphs, as depicted in Figure 4 and Figure 6. Of note, in the loss graphs depicted in Figures 3, 6, and 5, we see an asymptotic convergence (within some neighborhood) of the losses of both the generator and discriminator towards zero. This however does not by itself constitute completion of the training process. The persistent fluctuations in both discriminator and generator loss graphs are indicators of the continuing adversarial contest that continues well past the point at which the graphs appear to have settled close to each other. Figure 7 clearly shows the adversariality between the discriminator and the generator. As the discriminator loss rises, the generator loss falls, and vice versa.

4 Discussion

Our results demonstrate that realistic OCT images of the retina can be generated using Generative Adversarial Networks. This has many potential applications in the field of ophthalmology and in medicine and healthcare in general. In the current paradigm, physicians are trained by seeing cases of patients with certain diseases. The principles governing the diagnosis and management of such patients are learned during care. However, for less common or rare diseases, physicians in training get very limited exposure, and as such limited training. Our GAN approach points to a means of augmenting case loads and experience where needed. Another potential application for this is in the simulation of patient responses to certain drugs or surgical procedures in development. With a realistic model of the human retina, one is able to perform increasingly realistic simulations. This holds promise to significantly accelerate the pace of research, discovery, and development of therapies.

Unlike with traditional supervised learning problems such as image classification, the endpoint at which training can be halted is less clear. It is a subjective decision based on similarity of the generated images to the training sample distribution which it seeks to sample. This implies a need for the programmer to have an understanding of the data and the native probability distribution of that data type. Hence domain specific knowledge and interdisciplinarity will be necessary requisites for moving this field of research forward.

5 Conclusion

Here, we presented the first reported application of generative adversarial networks for the synthesis of OCT images of the retina. Our approach yielded realistic images depicting various retinal diseases such as macular holes, exudative macular degeneration, pathological myopic myopic, central serous retinopathy, and macula-off retinal detachments amongst others. This result opens up an avenue to several applications including for the acceleration of pharmacological and surgical therapies.

Acknowledgement

The author thanks Daniel T. Chang of IBM (retired) for endorsement to the arXiv’s Computer Science Artificial Intelligence Section. And he thanks Google Cloud Program for Startups for providing the computational resources to RETINA-AI Health, Inc for this study.

References

  • [1] T. Baba, A. Hirose, M. Moriyama, and M. Mochizuki. Tomographic image and visual recovery of acute macula-off rhegmatogenous retinal detachment. Graefe’s Archive for Clinical and Experimental Ophthalmology, 242(7):576–581, 2004.
  • [2] T. Bek and M. Kandi. Quantitative anomaloscopy and optical coherence tomography scanning in central serous chorioretinopathy. Acta Ophthalmologica Scandinavica, 78(6):632–637, 2000.
  • [3] S. E. Benson, P. G. Schlottmann, C. Bunce, W. Xing, and D. G. Charteris. Optical coherence tomography analysis of the macula after scleral buckle surgery for retinal detachment. Ophthalmology, 114(1):108–112, 2007.
  • [4] PM Burlina, N Joshi, KD Pacheco, TA Liu, and NM Bressler. Assessment of deep generative models for high-resolution synthetic retinal image generation of age-related macular degeneration. JAMA ophthalmology, 2019.
  • [5] F. Coscas, G. Coscas, E. Souied, S. Tick, and G. Soubrane. Optical coherence tomography identification of occult choroidal neovascularization in age-related macular degeneration. American Journal of Ophthalmology, 144(4):592–599, 2007.
  • [6] P. Costa, A. Galdran, M. I. Meyer, M. Niemeijer, M. Abràmoff, A. M. Mendonça, and A. Campilho. End-to-end adversarial retinal image synthesis. IEEE transactions on medical imaging, 37(3):781–791, 2018.
  • [7] A. Diaz-Pinto, A. Colomer, V. Naranjo, S. Morales, Y. Xu, and A. F. Frangi. Retinal image synthesis for glaucoma assessment using dcgan and vae models. In International Conference on Intelligent Data Engineering and Automated Learning, pages 224–232. Springer, 2018.
  • [8] H. Fujimoto, F. Gomi, T. Wakabayashi, M. Sawa, M. Tsujikawa, and Y. Tano. Morphologic changes in acute central serous chorioretinopathy evaluated by fourier-domain optical coherence tomography. Ophthalmology, 115(9):1494–1500, 2008.
  • [9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680, 2014.
  • [10] M. R. Hee, C. A. Puliafito, C. Wong, E. Reichel, J. S. Duker, J. S. Schuman, E. A. Swanson, and J. G. Fujimoto. Optical coherence tomography of central serous chorioretinopathy. American Journal of Ophthalmology, 120(1):65–74, 1995.
  • [11] Michael R Hee, Caroline R Baumal, Carmen A Puliafito, Jay S Duker, Elias Reichel, Jason R Wilkins, Jeffery G Coker, Joel S Schuman, Eric A Swanson, and James G Fujimoto. Optical coherence tomography of age-related macular degeneration and choroidal neovascularization. Ophthalmology, 103(8):1260–1270, 1996.
  • [12] Puliafito C. A. Duker J. S. Reichel E. Coker J. G. Wilkins J. R. Schuman J. S. Swanson E. A. Hee, M. R. and J. G. Fujimoto. Topography of diabetic macular edema with optical coherence tomography. Ophthalmology, 105(2):360–370, 1998.
  • [13] Puliafito C. A. Wong C. Duker J. S. Reichel E. Rutledge B. Schuman J. S. Swanson E. A. Hee, M. R. and J. G. Fujimoto. Quantitative assessment of macular edema with optical coherence tomography. Archives of Ophthalmology, 113(8):1019–1029, 1995.
  • [14] David Huang, Eric A Swanson, Charles P Lin, Joel S Schuman, William G Stinson, Warren Chang, Michael R Hee, Thomas Flotte, Kenton Gregory, Carmen A Puliafito, et al. Optical coherence tomography. Science, 254(5035):1178–1181, 1991.
  • [15] T. Iida, N. Hagimura, T. Sato, and S. Kishi. Evaluation of central serous chorioretinopathy with optical coherence tomography. American Journal of Ophthalmology, 129(1):16–20, 2000.
  • [16] P. A. Keane, P. J. Patel, S. Liakopoulos, F. M. Heussen, S. R. Sadda, and A. Tufail. Evaluation of age-related macular degeneration with optical coherence tomography. Survey of ophthalmology, 57(5):389–414, 2012.
  • [17] S. J. Kim, M. Belair, N. M. Bressler, J. P. Dunn, J. E. Thorne, S. R. Kedhar, and D. A. Jabs. A method of reporting macular edema after cataract surgery using optical coherence tomography. Retina, 28(6):870–876, 2008.
  • [18] S. J. Kim, R. Equi, and N. M. Bressler. Analysis of macular edema after cataract surgery in patients with diabetes using optical coherence tomography. Ophthalmology, 114(5):881–889, 2007.
  • [19] Xiaoming Liu, Tianyu Fu, Zhifang Pan, Dong Liu, Wei Hu, and Bo Li. Semi-supervised automatic layer and fluid region segmentation of retinal optical coherence tomography images using adversarial learning. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 2780–2784. IEEE, 2018.
  • [20] P. Massin, G. Duguid, A. Erginay, B. Haouchine, and A. Gaudric. Optical coherence tomography for evaluating diabetic macular edema before and after vitrectomy. American journal of ophthalmology, 135(2):169–177, 2003.
  • [21] J. A. Montero and J. M. Ruiz-Moreno. Optical coherence tomography characterisation of idiopathic central serous chorioretinopathy. British Journal of Ophthalmology, 89(5):562–564, 2005.
  • [22] H. Nakanishi, M. Hangai, N. Unoki, A. Sakamoto, A. Tsujikawa, M. Kita, and N. Yoshimura. Spectral-domain optical coherence tomography imaging of the detached macula in rhegmatogenous retinal detachment. Retina, 29(2):232–242, 2009.
  • [23] T. Otani, S. Kishi, and Y. Maruyama. Patterns of diabetic macular edema with optical coherence tomography. American journal of ophthalmology, 127(6):688–693, 1999.
  • [24] I. Perente, C. A. Utine, C. Ozturker, M. Cakir, V. Kaya, H. Eren, Z. Kapran, and O. F. Yilmaz. Evaluation of macular changes after uncomplicated phacoemulsification surgery by optical coherence tomography. Current Eye Research, 32(3):241–247, 2007.
  • [25] C. A. Puliafito, M. R. Hee, C. P. Lin, E. Reichel, J. S. Schuman, J. S. Duker, J. A. Izatt, E. A. Swanson, and J. G. Fujimoto. Imaging of macular diseases with optical coherence tomography. Ophthalmology, 102(2):217–229, 1995.
  • [26] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
  • [27] P. J. Rosenfeld, A. E. Fung, and C. A. Puliafito. Optical coherence tomography findings after an intravitreal injection of bevacizumab (avastin®) for macular edema from central retinal vein occlusion. Ophthalmic Surgery, Lasers and Imaging Retina, 36(4):336–339, 2005.
  • [28] Jürgen Schmidhuber. Learning factorial codes by predictability minimization. Neural Computation, 4(6):863–879, 1992.
  • [29] P. Sourdille and P. Santiago. Optical coherence tomography of macularthickness after cataract surgery. Journal of Cataract & Refractive Surgery, 25(2):256–261, 1999.
  • [30] E. A. Swanson, J. A. Izatt, M. R. Hee, D. Huang, C. P. Lin, J. S. Schuman, C. A. Puliafito, and J. G. Fujimoto. In vivo retinal imaging by optical coherence tomography. Optics letters, 18(21):1864–1866, 1993.
  • [31] R. Tennakoon, A. K. Gostar, R. Hoseinnezhad, and A. Bab-Hadiashar.

    Retinal fluid segmentation in oct images using adversarial loss based convolutional neural networks.

    In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pages 1436–1440. IEEE, 2018.
  • [32] B. von Jagow, C. Ohrloff, and T. Kohnen. Macular thickness after uneventful cataract surgery determined by optical coherence tomography. Graefe’s Archive for Clinical and Experimental Ophthalmology, 245(12):1765–1771, 2007.
  • [33] T. J. Wolfensberger. Foveal reattachment after macula-off retinal detachment occurs faster after vitrectomy than after buckle surgery. Ophthalmology, 111(7):1340–1343, 2004.
  • [34] T. J. Wolfensberger and M. Gonvers. Optical coherence tomography in the evaluation of incomplete visual acuity recovery after macula-off retinal detachments. Graefe’s Archive for Clinical and Experimental Ophthalmology, 240(2):85–89, 2002.