share
1 Introduction
Optical Coherence Tomography (OCT)[14]
has revolutionized the field of retina. And usage of this technology has become standard in the management of the vast majority of retinal patient encounters. An understanding of the native probability distribution of OCT representation of retinal disease is therefore essential. This is especially so as we seek to acquire more personalized understanding of disease, so as to develop treatments that are most fitting to the pathology of interest.
Following the work of Goodfellow et al[9], Generative Adversarial Networks (GANs), an adversarial type of generative model, has gained popularity based on the increasingly realistic datasets it can generate. Some theory of adversarial training algorithms has been in existence for at least 3 decades (Schmidhuber, 1992)[28]. However, practical feasibility, awareness, traction, and performance are only now notably rising. Interesting applications have included ”deep fakes” of human faces, and more recently, of medical data. For instance, Burlina et al [4] recently used GAN architecture to synthesize fundus images of age-related macular degeneration.
To our knowledge, this is the first use of generative adversarial networks to perform end-to-end synthesis OCT images of the retina. GANs have recently been used for the segmentation of various anatomical parts of the eye within ophthalmic images [31, 19], as well as conditional adversarial generation from vessel trees [6]. They have also been used to generate fundus photographs of the retina [7].
The capability to sample never before seen or even not yet existing retinal pathology will increase both our understanding of disease and facilitate development and assessment of new therapies. One such pathological feature which our GAN architecture detected is macular edema as well as subretinal fluid. Firstly, the presence of macular edema or subretinal fluid in the macula is essentially always indicative of pathology. It is a canonical hallmark that underlies several common retinal conditions such as diabetic macular edema[12, 20, 13, 23, 25], exudative macular degeneration [11, 14, 30, 5, 16], retinal vein occlusions [27], pseudophakic macular edema [32, 29, 17, 18, 24] central serous chorioretinopathy [10, 15, 2, 21, 8], and macula-off retinal detachments [34, 1, 3, 33, 22]. And therefore having a fuller understanding of all the ways and forms of macular edema will be essential to advancing our ability to develop effective therapies for a great number of retinal diseases.
In the remainder of this paper, we present our methods, results, and a discussion of the implications of this development for retinal research.
2 Methods
AI Generated images. Sampled from Epoch 96000 to 99800
A Generative Adversarial Network was implemented in Keras. The algorithm was run on an 8 core NVIDIA V100 machine equivalent with 600 GB of RAM. A database consisting of 500,000 images after augmentation was utilized for the training. The database had a diverse representation of pathology including macular holes, cystoid macular edema, exudative and non-exudative macular degeneration, central serous retinopathy, macula-off retinal detachments, and normal retina. Training was done for 100,000 epochs. A randomized vector was deconvolved according to the architecture depicted in Figure 1, yielding the generator output. The deconvolutional neural network weights were iteratively trained via backpropagation as directed by its loss function, a binary cross entropy. Our GAN architecture had some similarity as well as some notable differences compared to that recommended heuristically by Radford et al
[26]. LeakyReLu was used for the activation function of all layers of both the generator and discrimator, except for their final layer where a tanh and a sigmoid was used respectively. Batchnormalization was done on every layer of the generator and on no layer of the discriminator.
3 Results
Figure 1 shows realistic OCT images of the retina which were synthesized after 99800 training epochs of the algorithm. At the onset, prior to training, the initial random vector of size 100 yielded Figure 2 from the generator. Figure 7 to Figure 8 depict the loss landscape of the discriminator and generator during training.
At the onset of training, where the discriminator is first getting trained, we see a decrease in the discriminator’s loss and a concurrent rise in the generator’s loss. This early behavior is even marked by an early crossing in the loss graphs, as depicted in Figure 4 and Figure 6. Of note, in the loss graphs depicted in Figures 3, 6, and 5, we see an asymptotic convergence (within some neighborhood) of the losses of both the generator and discriminator towards zero. This however does not by itself constitute completion of the training process. The persistent fluctuations in both discriminator and generator loss graphs are indicators of the continuing adversarial contest that continues well past the point at which the graphs appear to have settled close to each other. Figure 7 clearly shows the adversariality between the discriminator and the generator. As the discriminator loss rises, the generator loss falls, and vice versa.
4 Discussion
Our results demonstrate that realistic OCT images of the retina can be generated using Generative Adversarial Networks. This has many potential applications in the field of ophthalmology and in medicine and healthcare in general. In the current paradigm, physicians are trained by seeing cases of patients with certain diseases. The principles governing the diagnosis and management of such patients are learned during care. However, for less common or rare diseases, physicians in training get very limited exposure, and as such limited training. Our GAN approach points to a means of augmenting case loads and experience where needed. Another potential application for this is in the simulation of patient responses to certain drugs or surgical procedures in development. With a realistic model of the human retina, one is able to perform increasingly realistic simulations. This holds promise to significantly accelerate the pace of research, discovery, and development of therapies.
Unlike with traditional supervised learning problems such as image classification, the endpoint at which training can be halted is less clear. It is a subjective decision based on similarity of the generated images to the training sample distribution which it seeks to sample. This implies a need for the programmer to have an understanding of the data and the native probability distribution of that data type. Hence domain specific knowledge and interdisciplinarity will be necessary requisites for moving this field of research forward.
5 Conclusion
Here, we presented the first reported application of generative adversarial networks for the synthesis of OCT images of the retina. Our approach yielded realistic images depicting various retinal diseases such as macular holes, exudative macular degeneration, pathological myopic myopic, central serous retinopathy, and macula-off retinal detachments amongst others. This result opens up an avenue to several applications including for the acceleration of pharmacological and surgical therapies.
Acknowledgement
The author thanks Daniel T. Chang of IBM (retired) for endorsement to the arXiv’s Computer Science Artificial Intelligence Section. And he thanks Google Cloud Program for Startups for providing the computational resources to RETINA-AI Health, Inc for this study.
References
- [1] T. Baba, A. Hirose, M. Moriyama, and M. Mochizuki. Tomographic image and visual recovery of acute macula-off rhegmatogenous retinal detachment. Graefe’s Archive for Clinical and Experimental Ophthalmology, 242(7):576–581, 2004.
- [2] T. Bek and M. Kandi. Quantitative anomaloscopy and optical coherence tomography scanning in central serous chorioretinopathy. Acta Ophthalmologica Scandinavica, 78(6):632–637, 2000.
- [3] S. E. Benson, P. G. Schlottmann, C. Bunce, W. Xing, and D. G. Charteris. Optical coherence tomography analysis of the macula after scleral buckle surgery for retinal detachment. Ophthalmology, 114(1):108–112, 2007.
- [4] PM Burlina, N Joshi, KD Pacheco, TA Liu, and NM Bressler. Assessment of deep generative models for high-resolution synthetic retinal image generation of age-related macular degeneration. JAMA ophthalmology, 2019.
- [5] F. Coscas, G. Coscas, E. Souied, S. Tick, and G. Soubrane. Optical coherence tomography identification of occult choroidal neovascularization in age-related macular degeneration. American Journal of Ophthalmology, 144(4):592–599, 2007.
- [6] P. Costa, A. Galdran, M. I. Meyer, M. Niemeijer, M. Abràmoff, A. M. Mendonça, and A. Campilho. End-to-end adversarial retinal image synthesis. IEEE transactions on medical imaging, 37(3):781–791, 2018.
- [7] A. Diaz-Pinto, A. Colomer, V. Naranjo, S. Morales, Y. Xu, and A. F. Frangi. Retinal image synthesis for glaucoma assessment using dcgan and vae models. In International Conference on Intelligent Data Engineering and Automated Learning, pages 224–232. Springer, 2018.
- [8] H. Fujimoto, F. Gomi, T. Wakabayashi, M. Sawa, M. Tsujikawa, and Y. Tano. Morphologic changes in acute central serous chorioretinopathy evaluated by fourier-domain optical coherence tomography. Ophthalmology, 115(9):1494–1500, 2008.
- [9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680, 2014.
- [10] M. R. Hee, C. A. Puliafito, C. Wong, E. Reichel, J. S. Duker, J. S. Schuman, E. A. Swanson, and J. G. Fujimoto. Optical coherence tomography of central serous chorioretinopathy. American Journal of Ophthalmology, 120(1):65–74, 1995.
- [11] Michael R Hee, Caroline R Baumal, Carmen A Puliafito, Jay S Duker, Elias Reichel, Jason R Wilkins, Jeffery G Coker, Joel S Schuman, Eric A Swanson, and James G Fujimoto. Optical coherence tomography of age-related macular degeneration and choroidal neovascularization. Ophthalmology, 103(8):1260–1270, 1996.
- [12] Puliafito C. A. Duker J. S. Reichel E. Coker J. G. Wilkins J. R. Schuman J. S. Swanson E. A. Hee, M. R. and J. G. Fujimoto. Topography of diabetic macular edema with optical coherence tomography. Ophthalmology, 105(2):360–370, 1998.
- [13] Puliafito C. A. Wong C. Duker J. S. Reichel E. Rutledge B. Schuman J. S. Swanson E. A. Hee, M. R. and J. G. Fujimoto. Quantitative assessment of macular edema with optical coherence tomography. Archives of Ophthalmology, 113(8):1019–1029, 1995.
- [14] David Huang, Eric A Swanson, Charles P Lin, Joel S Schuman, William G Stinson, Warren Chang, Michael R Hee, Thomas Flotte, Kenton Gregory, Carmen A Puliafito, et al. Optical coherence tomography. Science, 254(5035):1178–1181, 1991.
- [15] T. Iida, N. Hagimura, T. Sato, and S. Kishi. Evaluation of central serous chorioretinopathy with optical coherence tomography. American Journal of Ophthalmology, 129(1):16–20, 2000.
- [16] P. A. Keane, P. J. Patel, S. Liakopoulos, F. M. Heussen, S. R. Sadda, and A. Tufail. Evaluation of age-related macular degeneration with optical coherence tomography. Survey of ophthalmology, 57(5):389–414, 2012.
- [17] S. J. Kim, M. Belair, N. M. Bressler, J. P. Dunn, J. E. Thorne, S. R. Kedhar, and D. A. Jabs. A method of reporting macular edema after cataract surgery using optical coherence tomography. Retina, 28(6):870–876, 2008.
- [18] S. J. Kim, R. Equi, and N. M. Bressler. Analysis of macular edema after cataract surgery in patients with diabetes using optical coherence tomography. Ophthalmology, 114(5):881–889, 2007.
- [19] Xiaoming Liu, Tianyu Fu, Zhifang Pan, Dong Liu, Wei Hu, and Bo Li. Semi-supervised automatic layer and fluid region segmentation of retinal optical coherence tomography images using adversarial learning. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 2780–2784. IEEE, 2018.
- [20] P. Massin, G. Duguid, A. Erginay, B. Haouchine, and A. Gaudric. Optical coherence tomography for evaluating diabetic macular edema before and after vitrectomy. American journal of ophthalmology, 135(2):169–177, 2003.
- [21] J. A. Montero and J. M. Ruiz-Moreno. Optical coherence tomography characterisation of idiopathic central serous chorioretinopathy. British Journal of Ophthalmology, 89(5):562–564, 2005.
- [22] H. Nakanishi, M. Hangai, N. Unoki, A. Sakamoto, A. Tsujikawa, M. Kita, and N. Yoshimura. Spectral-domain optical coherence tomography imaging of the detached macula in rhegmatogenous retinal detachment. Retina, 29(2):232–242, 2009.
- [23] T. Otani, S. Kishi, and Y. Maruyama. Patterns of diabetic macular edema with optical coherence tomography. American journal of ophthalmology, 127(6):688–693, 1999.
- [24] I. Perente, C. A. Utine, C. Ozturker, M. Cakir, V. Kaya, H. Eren, Z. Kapran, and O. F. Yilmaz. Evaluation of macular changes after uncomplicated phacoemulsification surgery by optical coherence tomography. Current Eye Research, 32(3):241–247, 2007.
- [25] C. A. Puliafito, M. R. Hee, C. P. Lin, E. Reichel, J. S. Schuman, J. S. Duker, J. A. Izatt, E. A. Swanson, and J. G. Fujimoto. Imaging of macular diseases with optical coherence tomography. Ophthalmology, 102(2):217–229, 1995.
- [26] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
- [27] P. J. Rosenfeld, A. E. Fung, and C. A. Puliafito. Optical coherence tomography findings after an intravitreal injection of bevacizumab (avastin®) for macular edema from central retinal vein occlusion. Ophthalmic Surgery, Lasers and Imaging Retina, 36(4):336–339, 2005.
- [28] Jürgen Schmidhuber. Learning factorial codes by predictability minimization. Neural Computation, 4(6):863–879, 1992.
- [29] P. Sourdille and P. Santiago. Optical coherence tomography of macularthickness after cataract surgery. Journal of Cataract & Refractive Surgery, 25(2):256–261, 1999.
- [30] E. A. Swanson, J. A. Izatt, M. R. Hee, D. Huang, C. P. Lin, J. S. Schuman, C. A. Puliafito, and J. G. Fujimoto. In vivo retinal imaging by optical coherence tomography. Optics letters, 18(21):1864–1866, 1993.
-
[31]
R. Tennakoon, A. K. Gostar, R. Hoseinnezhad, and A. Bab-Hadiashar.
Retinal fluid segmentation in oct images using adversarial loss based convolutional neural networks.
In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pages 1436–1440. IEEE, 2018. - [32] B. von Jagow, C. Ohrloff, and T. Kohnen. Macular thickness after uneventful cataract surgery determined by optical coherence tomography. Graefe’s Archive for Clinical and Experimental Ophthalmology, 245(12):1765–1771, 2007.
- [33] T. J. Wolfensberger. Foveal reattachment after macula-off retinal detachment occurs faster after vitrectomy than after buckle surgery. Ophthalmology, 111(7):1340–1343, 2004.
- [34] T. J. Wolfensberger and M. Gonvers. Optical coherence tomography in the evaluation of incomplete visual acuity recovery after macula-off retinal detachments. Graefe’s Archive for Clinical and Experimental Ophthalmology, 240(2):85–89, 2002.
Comments
There are no comments yet.