CS767 Final Project
Despite the recent advances in automatically describing image contents, their applications have been mostly limited to image caption datasets containing natural images (e.g., Flickr 30k, MSCOCO). In this paper, we present a deep learning model to efficiently detect a disease from an image and annotate its contexts (e.g., location, severity and the affected organs). We employ a publicly available radiology dataset of chest x-rays and their reports, and use its image annotations to mine disease names to train convolutional neural networks (CNNs). In doing so, we adopt various regularization techniques to circumvent the large normal-vs-diseased cases bias. Recurrent neural networks (RNNs) are then trained to describe the contexts of a detected disease, based on the deep CNN features. Moreover, we introduce a novel approach to use the weights of the already trained pair of CNN/RNN on the domain-specific image/text dataset, to infer the joint image/text contexts for composite image labeling. Significantly improved image annotation results are demonstrated using the recurrent neural cascade model by taking the joint image/text contexts into account.READ FULL TEXT VIEW PDF
We propose and demonstrate a novel machine learning algorithm that asses...
Hand-crafted features based on linguistic and domain-knowledge play cruc...
Chest X-rays are one of the most common radiological examinations in dai...
This paper describes a novel text-to-speech (TTS) technique based on dee...
To develop a high throughput multi-label annotator for body Computed
In this work, we exploit the task of joint classification and weakly
While initially devised for image categorization, convolutional neural
CS767 Final Project
Comprehensive image understanding requires more than single object classification. There have been many advances in automatic generation of image captions to describe image contents, which is closer to a more complete image understanding than classifying an image to a single object class. Our work is inspired by many of the recent progresses in image caption generation[44, 54, 36, 14, 61, 15, 6, 62, 31], as well as some of the earlier pioneering work [39, 17, 16]
. The former have substantially improved performance, largely due to the introduction of ImageNet database and to advances in deep convolutional neural networks (CNNs), effectively learning to recognize the images with a large pool of hierarchical representations. Most recent work also adapt recurrent neural networks (RNNs), using the rich deep CNN features to generate image captions. However, the applications of the previous studies were limited to natural image caption datasets such as Flickr8k , Flickr30k , or MSCOCO  which can be generalized from ImageNet.
Likewise, there have been continuous efforts and progresses in the automatic recognition and localization of specific diseases and organs, mostly on datasets where target objects are explicitly annotated [50, 26, 55, 46, 40, 49]. Yet, learning from medical image text reports and generating annotations that describe diseases and their contexts have been very limited. Nonetheless, providing a description of a medical image’s content similar to a radiologist would describe could have a great impact. A person can better understand a disease in an image if it is presented with its context, e.g., where the disease is, how severe it is, and which organ is affected. Furthermore, a large collection of medical images can be automatically annotated with the disease context and the images can be retrieved based on their context, with queries such as “find me images with pulmonary disease in the upper right lobe”.
In this work, we demonstrate how to automatically annotate chest x-rays with diseases along with describing the contexts of a disease, e.g., location, severity, and the affected organs. A publicly available radiology dataset is exploited which contains chest x-ray images and reports published on the Web as a part of the OpenI  open source literature and biomedical image collections. An example of a chest x-ray image, report, and annotations available on OpenI is shown in Figure 1.
A common challenge in medical image analysis is the data bias. When considering the whole population, diseased cases are much rarer than healthy cases, which is also the case in the chest x-ray dataset used. Normal cases account for 37% (2,696 images) of the entire dataset (7,284 images), compared to the most frequent disease111Clinical findings, disorders, and other abnormal artifacts will be collectively referred to as diseases in this paper. case “opacity” which accounts for 12% (840 images) and the next frequent “cardiomegaly” constituting for 9% (655 images). In order to circumvent the normal-vs-diseased cases bias, we adopt various regularization techniques in CNN training.
In analogy to the previous works using ImageNet-trained CNN features for image encoding and RNNs to generate image captions, we first train CNN models with one disease label per chest x-ray inferred from image annotations, e.g., “calcified granuloma”, or “cardiomegaly”. However, such single disease labels do not fully account for the context of a disease. For instance, “calcified granuloma in right upper lobe” would be labeled the same as the “small calcified granuloma in left lung base” or “multiple calcified granuloma”.
Inspired by the ideas introduced in [28, 64, 27, 62, 60], we employ the already trained RNNs to obtain the context of annotations, and recurrently use this to infer the image labels with contexts as attributes. Then we re-train CNNs with the obtained joint image/text contexts and generate annotations based on the new CNN features. With this recurrent cascade model, image/text contexts are taken into account for CNN training (images with “calcified granuloma in right upper lobe” and “small calcified granuloma in left lung base” will be assigned different labels), to ultimately generate better and more accurate image annotations.
to combine recent advances in computer vision and machine translation. We also exploit the concepts of leveraging the mid-level RNN representations to infer image labels from the annotations[28, 64, 27, 62, 60].
Methods for mining and predicting labels from radiology images and reports were investigated in [51, 52, 57]. However, the image labels were mostly limited to disease names and did not contain much contextual information. Furthermore, the majority of cases in the datasets were diseased cases. In reality, most cases are normal, so that it is a challenge to detect relatively rarer diseased cases within such unbalanced data.
Mining images and image labels from a large collections of photo streams and blog posts on the Web were demonstrated in [34, 33, 35] where images could be searched with natural language queries. Associating neural word embeddings and deep image representations were explored in , but generating descriptions from such images/text pairs or image/word embeddings have not yet been demonstrated.
Detecting diseases from x-rays was demonstrated in [3, 45, 29], classifying chest x-ray image views in , and segmenting body parts in chest x-rays and computed tomography in [5, 21]. However, learning image contexts from text and re-generating image descriptions similar to what a human would describe have not yet been studied. To the best of our knowledge, this is the first study mining from a radiology image and report dataset, not only to classify and detect images but also to describe their context.
We use a publicly available radiology dataset of chest x-rays and reports that is a subset of the OpenI  open source literature and biomedical image collections. It contains 3,955 radiology reports from the Indiana Network for Patient Care, and 7,470 associated chest x-rays from the hospitals’ picture archiving systems. The entire dataset has been fully anonymized via an aggressive anonymization scheme, which achieved 100% precision in de-identification. However, a few findings have been rendered uninterpretable. More details about the dataset and the anonymization procedure can be found in , and an example case of the dataset is shown in Figure 1.
Each report is structured as comparison, indication, findings, and impression sections, in line with a common radiology reporting format for diagnostic chest x-rays. In the example shown in Figure 1, we observe an error resulting from the aggressive automated de-identification scheme. A word possibly indicating a disease was falsely detected as a personal information, and was thereby “anonymized” as “XXXX”. While radiology reports contain comprehensive information about the image and the patient, they may also contain information that cannot be inferred from the image content. For instance, in the example shown in Figure 1
, it is probably impossible to determine that the image is of a Burmese male.
On the other hand, a manual annotation of MEDLINE ® citations with controlled vocabulary terms (Medical Subject Headings (MeSH®) 
) is known to significantly improve the quality of the image retrieval results[20, 22, 10]. MeSH terms for each radiology report in OpenI (available for public use) are annotated according to the process described in . We use these to train our model.
Nonetheless, it is impossible to assign a single image label based on MeSH and train a CNN to reproduce them, because MeSH terms seldom appear individually when describing an image. The twenty most frequent MeSH terms appear with other terms in more than 60% of the cases. Normal cases (term “normal”) on the contrary, do not have any overlap, and account for 37% of the entire dataset. The thirteen most frequent MeSH terms appearing more than 180 times are provided in Table 1, along with the total number of cases in which they appear, the number of cases they overlap with in an image and the overlap percentages. The x-ray images are provided in Portable Network Graphics (PNG) format, with sizes varying from to . We rescale all CNN input training and testing images to a size of .
|MeSH Term||Total||Overlap||Overlap Percent|
The CNN-RNN based image caption generation approaches [44, 54, 36, 14, 61, 15, 6, 62, 31] require a well-trained CNN to encode input images effectively. Unlike natural images that can simply be encoded by ImageNet-trained CNNs, chest x-rays differ significantly from the ImageNet images. In order to train CNNs with chest x-ray images, we sample some frequent annotation patterns with less overlaps for each image, in order to assign image labels to each chest x-ray image and train with cross-entropy criteria. This is similar to the previous works from [51, 52, 57], which mines disease labels of images from their annotation text (radiology reports).
We find 17 unique patterns of MeSH term combinations appearing in 30 or more cases. This allows us to split the dataset in training/validation/testing cases as 80%/10%/10% and place at least 10 cases each in the validation and testing sets. They include the terms shown in Table 1, as well as scoliosis, osteophyte, spondylosis, fractures/bone. MeSH terms appearing frequently but without unique appearance patterns include pulmonary atelectasis, aorta/tortuous, pleural effusion, cicatrix, etc. They often appear with other disease terms (e.g. consolidation, airspace disease, atherosclerosis). We retain about 40% of the full dataset with this disease image label mining, where the annotations for the remaining 60% of images are more complex (and it is therefore difficult to assign a single disease label).
We use the aforementioned 17 unique disease annotation patterns (in Table 1, and scoliosis, osteophyte, spondylosis, fractures/bone) to label the images and train CNNs. For this purpose, we adopt various regularization techniques to deal with the normal-vs-diseased cases bias. For our default CNN model we chose the simple yet effective Network-In-Network (NIN)  model because the model is small in size, fast to train, and achieves similar or better performance to the most commonly used AlexNet model . We then test whether our data can benefit from a more complex state-of-the-art CNN model, i.e. GoogLeNet .
From the 17 chosen disease annotation patterns, normal cases account for 71% of all images, well above the numbers of cases for the remaining 16 disease annotation patterns. We balance the number of samples for each case by augmenting the training images of the smaller cases where we randomly crop size images from the original size image.
Even when we balance the dataset by augmenting many diseased samples, it is difficult for a CNN to learn a good model to distinguish many diseased cases from normal cases which have many variations on their original samples. It was shown in  that normalizing via mini-batch statistics during training can serve as an effective regularization technique to improve the performance of a CNN model. By normalizing via mini-batch statistics, the training network was shown not to produce deterministic values for a given training example, thereby regularizing the model to generalize better.
Inspired by this and by the concept of Dropout , we regularize the normal-vs-diseased cases bias via randomly dropping out an excessive proportion of normal cases compared to the frequent diseased pattern when sampling mini-batches. We then normalize according to the mini-batch statistics where each mini-batch consists of a balanced number of samples per disease case and a random selection of normal case samples. The number of samples for disease cases is balanced by random cropping during training, where each image of a diseased case is augmented at least four times.
We test both regularization techniques to assess their effectiveness on our dataset. The training and validation accuracies of the NIN model with batch-normalization, data-dropout, and both are provided in Table2. While batch-normalization and data-dropout alone do not significantly improve performance, combining both increases the validation accuracy by about 2%.
|training accuracy||validation accuracy|
|NIN with batch-normalization (BN)||94.06%||56.65%|
|NIN with data-dropout (DDropout)||98.78%||58.38%|
|NIN with BN and DDropout||100.0%||62.43%|
|training accuracy||validation accuracy|
|GoogLeNet with BN and DDropout||98.11%||66.40%|
|GoogLeNet with BN, DDropout, No-Crop||100.0%||69.84%|
We also validate whether the dataset can benefit from a more complex GoogLeNet , which is arguably the current state-of-the-art CNN architecture.
We apply both batch-normalization and data-dropout, and follow recommendations suggested in  (where human accuracy on the ImageNet dataset is superseded): increase learning rate, remove dropout, remove local response normalization.
The final training and validation accuracies using GoogLeNet model are provided in Table 3, where we achieve a higher (%) accuracy222The final models are trained with default learning rate of , with step down learning rate scheduling decreasing the learning rate by 50% and 33% each for NIN and GoogLeNet model in every 1/3th of the total training epochs.
We could not achieve high enough validation accuracy using exponential learning rate decay as in
training epochs. We could not achieve high enough validation accuracy using exponential learning rate decay as in.. We also observe a further % increase in accuracy when the images are no longer cropped, but merely duplicated to balance the dataset.
We use recurrent neural networks (RNNs) to learn the annotation sequence given input image CNN embeddings. We test both Long Short-Term Memory (LSTM)  and Gated Recurrent Unit (GRU)  implementations of RNNs. Simplified illustrations of LSTM and GRU are shown in Figure 2, and the details of both RNN implementations are briefly introduced below.
The LSTM implementation, originally proposed in , has been successfully applied to speech recognition , sequence generation , machine translation [7, 56, 43], and several image caption generation works mentioned in the main paper. LSTM is defined by the following equations:
where is the input gate, the forget gate, the output gate,
the state vector (memory),the new state vector (new memory), and the output vector. is a matrix of trained parameters (weights), and
is the logistic sigmoid function.represents the product of a vector with a gate value.
Please note that the notation used for the memory () and output () vectors differs from that in  and the other previous work. Our notation is intended to simplify the annotations and figures comparing LSTM to GRU.
The GRU implementation has been proposed most recently in , where it was successfully applied to machine translation. GRU is defined by the following equations:
where is the update gate, the reset gate, the new state vector, and the final state vector.
The number of MeSH terms describing diseases ranges from 1 to 8 (except normal
which is one word), with a mean of 2.56 and standard deviation of 1.36. The majority of descriptions contain up to five words. Since only 9 cases have images with descriptions longer than 6 words, we ignore these by constraining the RNNs to unroll up to 5 time steps. We zero-pad annotations with less than five words with the end-of-sentence token to fill in the five word space.
The parameters of the gates in LSTM and GRU decide whether to update their current state to the new candidate state , where these are learned from the previous input sequences. Further details about LSTM can be found in [24, 15, 14, 61], and about GRU and its comparisons to LSTM in [7, 30, 9, 8, 32]. We set the initial state of RNNs as the CNN image embedding (CNN()), and the first annotation word as the initial input. The output of the RNNs are the following annotation word sequences, and we train RNNs by minimizing the negative log likelihood of output sequences and true sequences:
where is the output word of RNN in time step , the correct word, CNN(I) the CNN embedding of input image , and the number of words in the annotation ( with the end-of-sequence zero-padding). Equation 11 is not a true conditional probability (because we only initialize the RNNs’ state vector to be CNN()) but a convenient way to describe the training procedure.
Unlike the previous work [31, 15, 14] where they use the last (FC-8) or second last (FC-7) fully-connected layers of AlexNet  or VGG-Net  model, the NIN or GoogLeNet models replace the fully-connected layers with the average-pooling layers [41, 58]. We therefore use the output of the last spatial average-pooling layer as the image embedding to initialize the RNN state vectors. The size of our RNNs’ state vectors are , which is identical to the output size of the average-pooling layers from NIN and GoogLeNet.
In sampling we again initialize the RNN state vectors with the CNN image embedding (=CNN()). We then use the CNN prediction of the input image as the first word as the input to the RNN, to sample following sequences up to five words. As previously, images are normalized by the batch statistics before being fed to the CNN. We use GoogLeNet as our default CNN model since it performs better than the NIN model in Sec. 5.2.
|BLEU -1/ -2/ -3 / -4||BLEU -1/ -2/ -3 / -4||BLEU -1/ -2/ -3 / -4|
|LSTM||82.6 / 19.2 / 2.2 / 0.0||67.4 / 15.1 / 1.6 / 0.0||78.3 / 0.4 / 0.0 / 0.0|
|GRU||98.9 / 46.9 / 1.2 / 0.0||85.8 / 14.1 / 0.0 / 0.0||75.4 / 3.0 / 0.0 / 0.0|
We evaluate the annotation generation on the BLEU 
score averaged over all of the images and their annotations in the training, validation, and test set. BLEU scores is a metric measuring a modified form of precision to compare n-gram words of generated and reference sentences. The BLEU scores evaluated are provided in Table4. The BLEU- scores are evaluated for cases with words in the annotations, using the implementation of .
We noticed that LSTM is easier to train, while GRU model yields better results with more carefully selected hyper-parameters333The final LSTM models are obtained with – learning rate: , learning rate decay: 0.97, decay rate: 0.95, without dropout; and the final GRU model is obtained with – learning rate: , learning rate decay: 0.99, decay rate: 0.99, with dropout rate: 0.9. With the same setting, adding dropout to LSTM model has adverse effect on its validation loss, similarly when increasing the number of LSTM layers to 3. The number of layers are 2 for both RNN models, and they are both trained with the batch size of 50.. While we find it difficult to conclude which model is better, the GRU model seems to achieve higher scores on average.
In Section 5, our CNN models are trained with disease labels only where the context of diseases are not considered. For instance, the same calcified granuloma label is assigned to all image cases that actually may describe the disease differently in a finer semantic level, such as “calcified granuloma in right upper lobe”, “small calcified granuloma in left lung base”, and “multiple calcified granuloma”.
Meanwhile, the RNNs trained in Section 6 encode the text annotation sequences given the CNN embedding of the image the annotation is describing. We therefore use the already trained CNN and RNN to infer better image labels, integrating the contexts of the image annotations beyond just the name of the disease. This is achieved by generating joint image/text context vectors that are computed by applying mean-pooling on the state vectors () of RNN at each step over the annotation sequence. Note, that the state vector of RNN is initialized with the CNN image embeddings (CNN()), and the RNN is unrolled over the annotation sequence, taking each word of the annotation as input. The procedure is illustrated in Figure 3, and the RNNs share the same parameters.
The obtained joint image/text context vector () encodes the image context as well as the text context describing the image. Using a notation similar to Equation 11, the joint image/text context vector can be written as:
where is the input word in the annotation sequence with words. Different annotations describing a disease are thereby separated into different categories by the , as shown in Figure 4. In Figure 4, the vectors of about fifty annotations describing calcified granuloma are projected onto two-dimensional planes via dimensionality reduction (), using the t-SNE  implementation of . We use the GRU implementation of the RNN because it showed better overall BLEU scores in Table 4. A visualization example for the annotations describing opacity can be found in the supplementary material.
From the generated for each of the image/annotation pair in the training and validation sets, we obtain new image labels taking disease context into account. In addition, we are no longer limited to disease annotation mostly describing a single disease. The joint image/text context vector summarizes both the image’s context and word sequence, so that annotations such as “calcified granuloma in right upper lobe”, “small calcified granuloma in left lung base”, and “multiple calcified granuloma” have different vectors based on their contexts.
Additionally, disease labels used in Section 5 with unique annotation patterns now have more cases, as cases with a disease described by different annotation words are no longer filtered out. For example, calcified granuloma previously had only 139 cases because cases with multiple diseases mentioned or with long description sequences were filtered out. At present, 414 cases are associated with calcified granuloma. Likewise, opacity now has 207 cases, as opposed to the previous 65. The average number of cases all first-mentioned disease labels has is , with a standard deviation of , a maximum of (calcified granuloma) and a minimum of (emphysema).
For a disease label having more than 170 cases ( (average+standard deviation)), we divide the cases into sub-groups of more than 50 cases by applying k-means clustering to the vector with . We train the CNN once more with the additional labels (, compared to in Section 5), train the RNN with the new CNN image embedding, and finally generate image annotations. The new RNN training cost function (compared to Equation 11) can be expressed as:
where denotes the joint image/text context vector obtained from the first round (with limited cases and image labels at 0th iteration) of CNN and RNN training. In the second CNN training round (1st iteration), we fine-tune from the previous , by replacing the last classification layer with the new set of labels () and training it with a lower learning rate (), except for the classification layer. The overall workflow is illustrated in Figure 5.
The final evaluated BLEU scores are provided in Table 5. We achieve better overall BLEU scores than those in Table 4 before using the joint image/text context. It is noticeable that higher BLEU-N () scores are achieved compared to Table 4, indicating that more comprehensive image contexts are taken into account for the CNN/RNN training. Also, slightly better BLEU scores are obtained using GRU on average and higher BLEU-1 scores are acquired using LSTM, although the comparison is empirical. Examples of generated annotations on the chest x-ray images are shown in Figure 6. These are generated using the GRU model, and more examples can be found in the supplementary material.
|BLEU -1/ -2/ -3 / -4||BLEU -1/ -2/ -3 / -4||BLEU -1/ -2/ -3 / -4|
|LSTM||97.2 / 67.1 / 14.9 / 2.8||68.1 / 30.1 / 5.2 / 1.1||79.3 / 9.1 / 0.0 / 0.0|
|GRU||89.7 / 61.7 / 28.5 / 11.0||61.9 / 29.6 / 11.3 / 2.0||78.5 / 14.4 / 4.7 / 0.0|
We present an effective framework to learn, detect disease, and describe their contexts from the patient chest x-rays and their accompanying radiology reports with Medical Subject Headings (MeSH) annotations. Furthermore, we introduce an approach to mine joint contexts from a collection of images and their accompanying text, by summarizing the CNN/RNN outputs and their states on each of the image/text instances. Higher performance on text generation is achieved on the test set if the joint image/text contexts are exploited to re-label the images and to train the proposed CNN/RNN framework subsequently.
To the best of our knowledge, this is the first study that mines from a publicly available radiology image and report dataset, not only to classify and detect disease in images but also to describe their context similar to a human observer would read. While we only demonstrate on a medical dataset, the suggested approach could also be applied to other application scenario with datasets containing co-existing pairs of images and text annotations, where the domain-specific images differ from those of the ImageNet.
This work was supported in part by the Intramural Research Program of the National Institutes of Health Clinical Center, and in part by a grant from the KRIBB Research Initiative Program (Korean Biomedical Scientist Fellowship Program), Korea Research Institute of Bioscience and Biotechnology, Republic of Korea. We thank NVIDIA for the K40 GPU donation.
Framing image description as a ranking task: Data, models and evaluation metrics.
Journal of Artificial Intelligence Research, 2013.
Addressing the rare word problem in neural machine translation.In Proceedings of ACL, 2015.
Scikit-learn: Machine learning in Python.Journal of Machine Learning Research, 2011.
Figure 8 shows the vectors of about fifty annotations describing calcified granuloma are projected onto two-dimensional planes via dimensionality reduction (), using the t-SNE  implementation of .
Figure 9 shows the visualization for the annotations describing opacity.
We can see that different annotations describing a disease are thereby separated into different categories by the .
More annotation generation examples are provided in Figures 10 and 11. Overall, the system generates promising results on predicting disease (labels) and its context (attributes) in the images. However, rare disease cases are more difficult to detect. For example, the cases pulmonary_atelectasis, spondylosis, and density (Figure 10), as well as foreign_bodies, atherosclerosis, costophrenic_angle, deformity (Figure 11) are much rarer in the data than calcified_granuloma, cardiomegaly, and all the frequent cases listed in Table 1 of the main paper.
Furthermore, the (left or right) location of the disease cannot be identified in a lateral view (obtained by scanning the patient from the side), as shown in Figure 11. Since our dataset contains a limited number of disease cases, we treat each x-ray image and report as a sample, and do not account for different views.
For future work, we plan to improve prediction accuracy by both (a) accounting for the different views, and (b) collecting a larger dataset to better account for rare diseases.