DeepAI
Log In Sign Up

UMLS-ChestNet: A deep convolutional neural network for radiological findings, differential diagnoses and localizations of COVID-19 in chest x-rays

06/06/2020
by   Germán González, et al.
0

In this work we present a method for the detection of radiological findings, their location and differential diagnoses from chest x-rays. Unlike prior works that focus on the detection of few pathologies, we use a hierarchical taxonomy mapped to the Unified Medical Language System (UMLS) terminology to identify 189 radiological findings, 22 differential diagnosis and 122 anatomic locations, including ground glass opacities, infiltrates, consolidations and other radiological findings compatible with COVID-19. We train the system on one large database of 92,594 frontal chest x-rays (AP or PA, standing, supine or decubitus) and a second database of 2,065 frontal images of COVID-19 patients identified by at least one positive Polymerase Chain Reaction (PCR) test. The reference labels are obtained through natural language processing of the radiological reports. On 23,159 test images, the proposed neural network obtains an AUC of 0.94 for the diagnosis of COVID-19. To our knowledge, this work uses the largest chest x-ray dataset of COVID-19 positive cases to date and is the first one to use a hierarchical labeling schema and to provide interpretability of the results, not only by using network attention methods, but also by indicating the radiological findings that have led to the diagnosis.

READ FULL TEXT VIEW PDF
04/30/2020

A Deep Convolutional Neural Network for COVID-19 Detection Using Chest X-Rays

We present an image classifier based on the CheXNet and a transfer learn...
12/26/2022

Diagnosis of COVID-19 based on Chest Radiography

The Coronavirus disease 2019 (COVID-19) was first identified in Wuhan, C...
01/22/2019

PadChest: A large chest x-ray image dataset with multi-label annotated reports

We present a labeled large-scale, high resolution chest x-ray dataset fo...
06/01/2020

BIMCV COVID-19+: a large annotated dataset of RX and CT images from COVID-19 patients

In this work we describe BIMCV-COVID-19+ dataset, a large dataset from M...
06/23/2021

CxSE: Chest X-ray Slow Encoding CNN forCOVID-19 Diagnosis

The coronavirus continues to disrupt our everyday lives as it spreads at...
11/01/2020

Triage of Potential COVID-19 Patients from Chest X-ray Images using Hierarchical Convolutional Networks

The current COVID-19 pandemic has motivated the researchers to use artif...

1 Introduction

The SARS-CoV2 pandemic has led researchers to develop deep learning methods on x-rays (CR/DX) and Computed Tomography (CT) images for the diagnosis of patients with COVID-19. While most methods are focused on the classification of images into the categories COVID-19, other pathologies or no pathology, the early detection and localization of lesions such as infiltrates, in particular ground glass opacities, is essential both for the diagnosis and to predict the evolution of the patient in order to help making clinical decisions.

The typical findings that suggest a COVID-19 infection are ground glass opacities that, even in the initial stages, affect both lungs, particularly the lower lobes, and especially the posterior segments, mainly with a peripheral and subpleural distribution  [26, 33, 19]. These findings were present in 44% of patients in the first two days on chest CT scans [2], in 75% of patients in the first four days [20] and in 86% of patients during illness days 0-5 [32].

Differential diagnosis
├── pneumonia
└── atypical pneumonia
└── viral pneumonia
├── COVID-19
Radiological Findings
├── infiltrates
├── interstitial pattern
├── ground glass pattern
└── alveolar pattern
├── consolidation
├── increased density
Differential diagnosis
├── pneumonia
└── atypical pneumonia
└── viral pneumonia
├── COVID-19
Radiological Findings
├── infiltrates
├── interstitial pattern
├── ground glass pattern
└── alveolar pattern
├── consolidation
├── increased density
Figure 1: Left: Example image of the BIMCV COVID-19 dataset. Middle: nodes of the UMLS hierarchy that are positive, as extracted from the radiology report, are marked in bold. Right: In bold, UMLS nodes positive obtained by the network. Please note that the network detects ground glass patterns and consolidations that are not present on the radiology report. Both radiological findings are highly related to the diagnosis of COVID-19.

Therefore, beyond classifying an image as COVID/No/Others, it is very important to obtain the radiological findings in order to understand the severity of the disease and to predict the possible evolution of the patient. Moreover, it should be taken into account that some PCR positive COVID-19 patients may show a normal RX, therefore only with the image information it may not be possible to perform a correct diagnosis. Similarly, not all radiological findings compatible with COVID-19, such as ground glass opacities, are indeed produced by a Sars-Cov-2 infection. Unlike diagnoses, findings are reported by the radiologists and they can be seen in the images, being less dependent on other clinical variables.

An example of the proposed methodology can be seen in Fig. 1. A chest x-ray (left) is tagged by processing its radiology report with natural language processing to a set of hierarchical labels (middle). The network predicts such labels that include not only the presence of the disease (COVID-19), but also the radiological findings that have lead to it. Please note that the model detects two findings that were not present on the radiology report but are consistent with COVID-19. The proposed method is not constrained to COVID-19, but contains an extense set of labels that encompasses most of the diseases and radiological findings that can be interpreted from a chest x-ray.

Deep learning methods are widely used for chest x-ray pathology detection. In order to train these models, it is essential to have a large dataset of images properly annotated. In the literature there are many datasets intended for this task such as [13, 14, 31], with a maximum of 14 output variables associated with them, and none with ground glass opacities or infiltrates as part of their covariates. To the best of our knowledge, the only open dataset currently labeled with a large set of radiological findings is PadChest [6]. PadChest includes more than 160,000 images obtained from 67,000 patients that were interpreted and reported by radiologists at San Juan Hospital (Spain) from 2009 to 2017, covering six different position views, and including additional information on image acquisition and patient demography. Padchest labels were extracted from the radiological reports, resulting in 22 differential diagnoses, 122 anatomic locations and 189 different radiological findings mapped onto standard Unified Medical Language System (UMLS) using controlled biomedical vocabulary unique identifiers (CUIs) that are organized into semantic hierarchical concept trees. Padchest is the only open dataset that provides labels for consolidations, pneumonia and ground glass opacities, as needed to diagnose COVID-19.

Different neural network architectures have been proposed in the literature for the classification of pathologies [23, 29] from RX images using the datasets from [13, 14, 31]. However, to the best of our knowledge, there is no previous work addressing the classification of a large set of radiological findings as described by [6].

Some recent models based on Convolutional Neural Networks (CNN) have been proposed for the detection of COVID 19. Most of them can be found in recent review articles, such as [25, 17, 5]. However, they all rely on the same public dataset [8] for positive samples [18, 12, 1], or subsequent works derived from it [15, 30, 11, 7]. The dataset from [8] is obtained by gathering images from other publications, which are indeed of low spatial and bitdepth resolution. As negative samples the cited work uses data from the RSNA challenge [22] or from the Kaggle competition [21], which is composed of pediatric images. Moreover, it should be considered that there is a domain mismatch between the positives and the negatives, and therefore, it may happen that models would find events on the images that are not related to the disease but on the domain itself.

2 Material and methods

2.1 Dataset

In this work we use the images and labels from PadChest [6] and BIMCV COVID-19+ [9]. Both contain labels that are mapped into NLM Unified Medical Language System (UMLS) controlled biomedical vocabulary unique identifiers, referred as CUIs, along with the localizations. Four additional labels with special coding rules were explicitly defined: normal, exclude, unchanged and suboptimal study. Although the excerpts from the report are provided in Spanish, labels are mapped onto CUI codes, thus making the dataset usable regardless of the language. In addition, the labels are structured using a tree structure.

Radiological Findings
├── infiltrates
├── interstitial pattern
├── ground glass pattern
├── reticular interstitial pattern
├── reticulonodular interstitial pattern
└── miliary opacities
└── alveolar pattern
├── consolidation
└── air bronchogram
└── air bronchogram
Differential diagnosis
├── pneumonia
└── atypical pneumonia
└── viral pneumonia
├── COVID-19
└── COVID-19 uncertain
├── tuberculosis
└── tuberculosis sequelae
├── lung metastasis
├── lymphangitis carcinomatosa
├── lepidic adenocarcinoma
Localization
├── extracorporal
├── cervical
├── soft tissue
├── subcutaneous
├── axilar
└── pectoral
└── nipple
├── bone
├── shoulder
├── acromioclavicular
Table 1: Examples of the hierarchical radiological findings, diagnoses and locations present on the reference standard. For the full tree we refer the reader to the supplementary material.

Some examples of the tree for differential diagnosis, radiological findings and localizations are shown in Table 1. Overall, PadChest contains more than 160,000 images with 189 different radiographic findings, 22 differential diagnoses and 122 anatomic locations. The 92,594 frontal images of the PadChest dataset were used for this work.

Since PadChest’s data were collected until 2017, it does not include COVID-19 labels. However, the recently released BIMCV COVID-19+ dataset, whose acquisition methodology and organization is based on PadChest, is intended for training machine learning methods to detect this pathology. This is currently the largest open chest X-ray COVID-19 dataset. It contains chest CXR (CR, DX) and computed tomography (CT) imaging of COVID-19 patients along with their radiographic findings, pathologies, Polymerase Chain Reaction (PCR), Immunoglobulin G (IgG) and Immunoglobulin M (IgM) diagnostic antibody tests, and radiological reports from Medical Imaging Databank in Valencian Region Medical Image Bank (BIMCV).

Like in PadChest, the BIMCV COVID-19+ labels cover a wide spectrum of thoracic entities, images are stored in high resolution and entities are localized with anatomical labels in a Medical Imaging Data Structure (MIDS) format. In addition, 18 image studies were annotated by a team of expert radiologists to include semantic segmentation of radiological findings. Moreover, it provides extensive information, including the patient’s demographic information, type of projection and acquisition parameters for the imaging study, among others.

The BIMCV COVID-19+ iteration used in the presented work includes 1,380 CR and 885 DX that were added to the full PadChest samples for training, testing and validation. COVID-19 is a pathology where RT-PCR is considered the gold standard for diagnosis of COVID-19 pneumonia [16]. Its pattern of radiological findings, even if typical, are not pathognomonic, i.e. their presence does not mean that COVID-19 pneumonia is present beyond any doubt and other pathologies as other atypical and viral pneumonias need to be ruled out. For this reason, COVID-19 pneumonia was included in the taxonomy tree under the differential diagnosis branch [9].

2.2 Image preprocesing

Only frontal images, either with PA or AP, supine, decubitus, lordotic or standing were included in this work. Such images were identified in the databases by inspecting the manual annotations. Images with photometric interpretation of MonochromeI

, as signaled on the DICOM fields were inverted. Image intensities were normalized by subtracting the mean dividing by the variance before using them to train any of the proposed networks.

In order to normalize the image size, we first cropped a centered square region of interest, with the size of the smallest image axis. Such technique is preferred to interpolating to a square grid in order to prevent anatomical deformations, and also preferred to image padding to prevent misleading the network at the radiography edges. Images were finally re-scaled to

pixels to be processed by the neural network.

2.3 Label propagation

The labels for each image are mapped to the UMLS hierarchy. They are extracted from the NLP interpretation of the radiological reports. Only few nodes of the UMLS tree are extracted from each radiology report. The UMLS hierarchy has an is-a relationship for both the differential diagnoses and radiological findings trees, and an is-part-of relationship for the anatomical location tree. We therefore construct for each image a target vector that places ’1’ on each node that is in the interpretation of the radiology report or any of their parents until the root. For instance, if the image is associated with a COVID-19 diagnosis as interpreted from the radiological report, and using the tree shown in Fig. 

1, we would flag the image as being positive for viral pneumonia, atopical pneumonia, pneumonia and differential diagnosis. An image with a label of pneumonia would not have a COVID-19 diagnosis.

2.4 Training

We evaluate three image classification topologies in this work: EfficientNetB4 [28], Inception-Resnet-V2 [27] and XNet [4]

. Each of these architectures is extended by using two fully connected layers of 512 units with rectified linear units (ReLU) activation and a final classification layer of length 306 with sigmoid activation. Dropout layers with dropout coefficient of

were interleaved between the fully connected layers. The model weights were pre-trained using the ImageNet

[10] database.

The loss function employed is a binary cross entropy for each of the output dimensions independently, since correlations between dimensions are already handled by the construction of the target labels through the label propagation method. The networks are trained using adaptive momentum optimization with a learning rate schedule that decreases the learning rate between

for the first epochs to

for the last ones. Training is performed for epochs and the model with the best validation accuracy was selected.

The training set consists of 50,000 frontal images from Padchest and 1,000 images from the BIMCV COVID-19+ dataset. 20,000 images from Padchest and 500 from COVID-19 were used for validation. The test set, from which the evaluation results are reported, is composed by 22,594 images from Padchest and 565 images from the BIMCV COVID-19+ dataset. The images are distributed in such manner that two images from the same patient can not appear in the training, validation or test sets.

2.5 Statistical Analysis

ROC curves were computed with Scikit-learn. 95% confidence intervals, when reported, were computed using MedCalc and with R statistical software.

3 Results

Variable N EfficientNetB4 InceptionResnetV2 XCeption
Avg AUC 23,159 0.787 (0.176) 0.738 (0.172) 0.768 (0.175)
pacemaker 367 0.997 (0.996-0.999) 0.997 (0.996-0.998) 0.997 (0.996-0.999)
single chamber device 69 0.992 (0.990-0.993) 0.989 (0.987-0.992) 0.991 (0.990-0.993)
sternotomy 382 0.998 (0.998-0.999) 0.969 (0.962-0.976) 0.996 (0.995-0.997)
dual chamber device 164 0.987 (0.975-0.999) 0.986 (0.975-0.997) 0.987 (0.975-0.999)
art. mitral heart valve 56 0.992 (0.987-0.996) 0.979 (0.971-0.988) 0.989 (0.983-0.995)
pulmonary edema 53 0.960 (0.945-0.976) 0.956 (0.943-0.969) 0.955 (0.939-0.971)
res. cent. ven. catheter 66 0.962 (0.938-0.987) 0.907 (0.870-0.945) 0.965 (0.941-0.989)
cavitation 77 0.942 (0.916-0.969) 0.939 (0.918-0.960) 0.951 (0.931-0.971)
….
mastectomy 109 0.811 (0.769-0.852) 0.656 (0.603-0.709) 0.748 (0.701-0.795)
diaphr. eventration 182 0.779 (0.748-0.809) 0.717 (0.684-0.750) 0.718 (0.685-0.751)
callus rib fracture 414 0.760 (0.740-0.781) 0.711 (0.689-0.733) 0.742 (0.720-0.764)
tracheal shift 148 0.763 (0.729-0.797) 0.658 (0.619-0.697) 0.790 (0.755-0.824)
air trapping 798 0.751 (0.734-0.768) 0.707 (0.690-0.725) 0.720 (0.703-0.738)
flattened diaphragm 136 0.743 (0.704-0.782) 0.704 (0.662-0.747) 0.717 (0.672-0.761)
meds. lipomatosis 76 0.713 (0.659-0.767) 0.688 (0.634-0.742) 0.653 (0.592-0.714)
azygos lobe 74 0.562 (0.504-0.619) 0.608 (0.548-0.667) 0.623 (0.566-0.680)
Table 2:

Performance of the three different networks in the top 8 and bottom 8 diagnoses and radiological findings. In parenthesis we show 95% confidence intervals. The average AUCs are obtained by averaging the AUCs of all predicted variables and are shown with standard deviations in parenthesis.

3.1 One class versus all

First, the performance of the method is evaluated in a per-label setting. As such, we consider as positive for a radiological finding, differential diagnosis or location, any image that has such variable or any of its children on the reference standard interpretation of the radiology report. This allows us to obtain a per variable positive and negative class. The comparison results of the three networks favors EfficientNetB4, with an average AUC over all radiological findings, locations and diagnoses of 0.787, in comparison with XCeption (AUC of 0.768) or InceptionResnetV2 (AUC of 0.737). Table 2 shows the top eight and bottom eight diagnoses or radiological findings detected by each network. In the supplementary material we show the performance for the whole UMLS tree.

3.2 COVID-19 Results

The aim of this work is to assist in the diagnosis of COVID-19. We therefore put special focus on the evaluation of those findings that are compatible with the disease. In chest x-rays, COVID-19 is associated with ground glass opacities, consolidations and alveolar patterns. Fig 2 shows the performance of the classifier for such variables. The area under the curve for the EfficientNetB4 network for COVID-19 is of 0.937 (95% CI between 0.924 to 0.950), for consolidations is of 0.924 (95% CI between 0.910 and 0.938), for ground glass opacities is of 0.840 (95% CI interval between 0.783 and 0.897), for alveolar patterns is of 0.923 (95% CI between 0.914 and 0.933) and for infiltrates of 0.871 (95% CI between 0.863 and 0.879).

Of interest is to show the capacity of the model to distinguish between COVID-19 and other pneumonias. To perform such test we have selected the images from the test set that have any type of pneumonia and evaluate in such subsets the performance of COVID-19 detection. The AUCs obtained are 0.73 for EfficientnetB4 and 0.70 for both InceptionResnetV2 and XCeption. When evaluating the EfficientNetB4, no image from padchest had a score for COVID-19 higher than 0.5, showing that the network does not produce false positives on COVID-19 negative images.

We have used the well known GradCam [24]

method to estimate the regions of the image that contributed most to the prediction of the variables. An example is shown in Fig. 

3. Consolidations appear mainly at the left lung, ground glass opacities at the right lung and the diagnosis of COVID 19 is produced by features located at both the left and right lungs, mainly on the lower lobes. While the three heatmaps are obtained independently, thy complement each other and explain the decisions made by the network.

The software was implemented with Keras and Tensorflow libraries in Python. Training was performed in parallel on four Nvidia 1080Ti GPUs cards. It took 11.4 minutes per epoch for the EfficientNetB4 network, 11 minutes per epoch for InceptionResnetV2 network and 21 minutes for XCeption.

Figure 2: Receiver operator characteristic curves for COVID-19 diagnosis and the radiological findings related to COVID-19 for the three network structures evaluated. Curves obtained on the test set. AUCs and 95% confidence intervals are shown as shaded areas and quantified in the legend.
Figure 3: GradCam network attention heatmaps for a subject with COVID 19 pneumonia. It is the same subject as Fig. 1. Please note how the COVID 19 diagnosis is related to bilateral lower attention, the consolidation is associated with mostly left (anatomically) attention, while the ground glass pattern is associated mainly with right attention.

4 Discussion

In this work, we present a method for chest x-ray image classification based on deep neural networks which outputs hierarchical radiological findings, differential diagnosis and localization outcome variables. The reference standard is obtained using natural language processing from the radiological reports.

On the 23,159 test images, the method shows a high average AUC score of all the variables (0.79 for EfficientNetB4), holding promise for its future use. Radiological findings that are large in size, specially due to artificial objects such as pacemakers or heart valves, are the ones showing higher AUCs. Smaller or subtle findings such as azygos lobe, rib fractures or air trapping show significantly lower performance. This could be partly explained by the image size that has been used in the method. Due to computational reasons, the image size had to be reduced to 299x299 pixels. Full-size images are around 8 megapixels. Such reduction in the image resolution eliminates textures and edges, that could be of great importance to detect subtle structures.

When evaluating the performance on COVID-19, the proposed method successfully distinguished COVID-19 patients, with an AUC of , showing promising results for the diagnosis of cases amidst the pandemic. When evaluated to distinguish between COVID-19 caused pneumonia and other pneumonia types, the AUC dropped to . When evaluating the network on the subset of patients with pneumonia from the PadChest dataset, no subject had a COVID-19 score higher than . The drop in the AUC can be explained by several reasons. The first one is the source of the reference standard. The BIMCV COVID-19+ dataset has been acquired amidst the beginning of the COVID-19 pandemic and the radiological reports may not have correctly included COVID-19 information. For instance, 10 out of the 18 image studies with regions of interest included in the dataset were not marked as being COVID-19 positive in the radiology report, although they have ground glass opacities or consolidations that were marked by reviewing radiologists specialized in COVID-19.

The reference standard used for training comes from natural language processing methods applied to the radiological reports. As mentioned by [3], such reference standard is considered of moderate quality

, and task-specific expert annotation is preferred for the training and development of artificial intelligence methods. The source of the reference standard is a limitation of the proposed work. It should be note that all prior art related to automated interpretation of chest x-rays using the datasets of 

[13, 6, 14, 31] have the same limitation. An evaluation of the quality of the reference standard would be necessary to solidify the results.

Further, the reference standard is composed of disease and locations, which are classified independently. It would be of interest to intertwine them, such that diseases that are in certain locations can not produce outcomes in incompatible locations and vice-versa. For instance, cardiomegaly as a single radiological finding can not appear at the same time as the location peripheral. Beyond locations, the hierarchical relationships of the variables are exploited at training, but not at detection, and they could be used to make stronger predictions. For instance, a moderately positive ground glass pattern could be rejected if their parent neurons did not fire adequately. Such methods are left as future work.

Different image classification networks can be used for the proposed methodology. In this work we have evaluated XCeption, InceptionResnetV2 and EFficientNetB4. The performance order followed that of the original networks on the ImageNet classification database, being EfficientNetB4 the network that has a highest average AUC. Still, the network and the training methodology chosen in this work are relatively simple for current state-of-the-art deep neural networks. Methods such as ensemble learning and cosine decay learning rate [15], uncertainty estimation  [12] or more advanced hierarchical training methods may improve the results.

A topic of discussion is whether the construction of the prediction labels by placing a ’1’ on the labels in the reference standard as well an all its parents is correct. For instance, while ground glass opacity (GGO) is an infiltrate, when describing the radiological findings in one image, one should clearly state that it is a GGO and not an infiltrate. We have addressed such topic in a two-fold manner. For training, we place ’1’ on all variables until the root. We do so since, using the same example as before, all GGO are indeed infiltrates. However, when we perform testing for GGO, all images that does not have GGO in the reference standard, even those with other kind of infiltrates, are marked as ’0’. This allows us to accurately evaluate the performance of GGO detection.

While the two datasets mixed for training are from the same region of Spain, they differ in their origin. Padchest is a single institution dataset (images were obtained from one hospital), while BIMCV COVID-19+ images were obtained from several institutions. This may lead to domain subtleties that are guiding the performance. A more in-depth evaluation of whether such is the case would be needed and is a limitation of this work. Furthermore, it should be considered that the majority of the population in the Valencian region is Caucasian.

Despite the above-mentioned limitations, the proposed work has value, since, to our knowledge, it is the first neural network used to interpret chest x-rays that uses hierarchical labels based on the UMLS entity relationships of radiological findings, differential diagnoses and locations. Further, it is the first work to use a large, open, dataset of COVID-19+ images for training and evaluation that comes from the same sources as the COVID-19 negative samples.

5 Conclusions

We have presented a method for the detection of hierarchical radiological findings (189 nodes), their anatomical location (122 nodes) and the diagnoses associated (22 variables) from chest x-rays. The method has been trained and evaluated on a composed large dataset of 94,659 frontal chest x-rays, achieving an average AUC of 0.787 for all the variables. When evaluated on COVID-19+ cases, results show a reliable performance, with an AUC of 0.94. The main advantage of the proposed method compared to previous works is that it provides explainability of the diagnoses, since not only the outcome variable are given, but also the radiological findings that have led to diagnoses, such as ground glass patterns (AUC of 0.84), consolidations (AUC of 0.92) or alveolar patterns (AUC of 0.92).

We believe that the proposed hierarchical labeling and training could be applied for further work on chest x-ray image interpretation. It is also planned to annotate COVID-19 related findings at a pixel level in order to perform semantic segmentation, since the number and area of these lesions is relevant both for the diagnosis and the prognostic.

Acknowledgments

This work has been supported by the grant awarded through decree 51/2020 by the Valencian Innovation Agency (Spain). This research is also supported by the project UACOVID19-18 from the University of Alicante. We thank also NVIDIA for the generous donation of a Titan Xp and a Quadro P6000 used in this research.

References

  • [1] I. D. Apostolopoulos and T. A. Mpesiana (2020)

    Covid-19: automatic detection from X-ray images utilizing transfer learning with convolutional neural networks

    .
    Physical and Engineering Sciences in Medicine (0123456789), pp. 1–6. External Links: Document, ISBN 0123456789, ISSN 2662-4729 Cited by: §1.
  • [2] A. Bernheim, X. Mei, M. Huang, Y. Yang, Z. Fayad, N. Zhang, K. Diao, B. Lin, X. Zhu, K. Li, S. Li, H. Shan, A. Jacobi, and M. Chung (2020-02) Chest ct findings in coronavirus disease-19 (covid-19): relationship to duration of infection. Radiology 295, pp. 200463. External Links: Document Cited by: §1.
  • [3] D. A. Bluemke, L. Moy, M. A. Bredella, B. B. Ertl-Wagner, K. J. Fowler, V. J. Goh, E. F. Halpern, C. P. Hess, M. L. Schiebler, and C. R. Weiss (2020) Assessing Radiology Research on Artificial Intelligence: A Brief Guide for Authors, Reviewers, and Readers-From the Radiology Editorial Board Key Considerations for Authors, Reviewers, and Readers of AI/ML Manuscripts in Radiology. Radiology 294. External Links: Document, Link Cited by: §4.
  • [4] J. Bullock, C. Cuesta-Lázaro, and A. Quera-Bofarull (2019) XNet: a convolutional neural network (cnn) implementation for medical x-ray image segmentation suitable for small datasets. In Medical Imaging 2019: Biomedical Applications in Molecular, Structural, and Functional Imaging, Vol. 10953, pp. 109531Z. Cited by: §2.4.
  • [5] A. Burlacu, R. Crisan-Dabija, I. V. Popa, B. Artene, V. Birzu, M. Pricop, C. Plesoianu, and D. Generali (2020) Curbing the ai-induced enthusiasm in diagnosing covid-19 on chest x-rays: the present and the near-future. medRxiv. Cited by: §1.
  • [6] A. Bustos, A. Pertusa, J. Salinas, and M. de la Iglesia-Vayá (2019) PadChest: a large chest x-ray image dataset with multi-label annotated reports. arXiv preprint arXiv:1901.07441. Cited by: §1, §1, §2.1, §4.
  • [7] M. E. Chowdhury, T. Rahman, A. Khandakar, R. Mazhar, M. A. Kadir, Z. B. Mahbub, K. R. Islam, M. S. Khan, A. Iqbal, N. Al-Emadi, et al. (2020) Can ai help in screening viral and covid-19 pneumonia?. arXiv preprint arXiv:2003.13145. Cited by: §1.
  • [8] J. P. Cohen, P. Morrison, and L. Dao (2020) COVID-19 Image Data Collection. External Links: 2003.11597 Cited by: §1.
  • [9] M. de la Iglesia Vayá, J. M. Saborit, J. A. Montell, A. Pertusa, A. Bustos, M. Cazorla, J. Galant, X. Barber, D. Orozco-Beltrán, F. Garcia, M. Caparrós, G. González, and J. M. Salinas (2020) BIMCV covid-19+: a large annotated dataset of rx and ct images from covid-19 patients. External Links: 2006.01174, Link Cited by: §2.1, §2.1.
  • [10] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei (2009) Imagenet: a large-scale hierarchical image database. In

    2009 IEEE conference on computer vision and pattern recognition

    ,
    pp. 248–255. Cited by: §2.4.
  • [11] M. Farooq and A. Hafeez (2020) COVID-ResNet: A Deep Learning Framework for Screening of COVID19 from Radiographs. External Links: 2003.14395, Link Cited by: §1.
  • [12] B. Ghoshal and A. Tucker (2020) Estimating Uncertainty and Interpretability in Deep Learning for Coronavirus (COVID-19) Detection. pp. 1–14. External Links: 2003.10769, Link Cited by: §1, §4.
  • [13] J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, et al. (2019) Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, pp. 590–597. Cited by: §1, §1, §4.
  • [14] A. E. Johnson, T. J. Pollard, S. Berkowitz, N. R. Greenbaum, M. P. Lungren, C. Deng, R. G. Mark, and S. Horng (2019) MIMIC-cxr: a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042. Cited by: §1, §1, §4.
  • [15] M. Karim, T. Döhmen, D. Rebholz-Schuhmann, S. Decker, M. Cochez, O. Beyan, et al. (2020) Deepcovidexplainer: explainable covid-19 predictions based on chest x-ray images. arXiv preprint arXiv:2004.04582. Cited by: §1, §4.
  • [16] H. Kim, H. Hong, and S. H. Yoon (2020) Diagnostic performance of ct and reverse transcriptase-polymerase chain reaction for coronavirus disease 2019: a meta-analysis. Radiology, pp. 201343. Cited by: §2.1.
  • [17] S. Latif, M. Usman, S. Manzoor, W. Iqbal, J. Qadir, G. Tyson, I. Castro, A. Razi, M. N. K. Boulos, A. Weller, et al. (2020)

    Leveraging data science to combat covid-19: a comprehensive review

    .
    Cited by: §1.
  • [18] X. Li, C. Li, and D. Zhu (2020) COVID-MobileXpert: On-Device COVID-19 Screening using Snapshots of Chest X-Ray. External Links: 2004.03042, Link Cited by: §1.
  • [19] M. Ng, E. Y. Lee, J. Yang, F. Yang, X. Li, H. Wang, M. M. Lui, C. S. Lo, B. Leung, P. Khong, et al. (2020) Imaging profile of the covid-19 infection: radiologic findings and literature review. Radiology: Cardiothoracic Imaging 2 (1), pp. e200034. Cited by: §1.
  • [20] F. Pan, T. Ye, P. Sun, S. Gui, B. Liang, L. Li, D. Zheng, J. Wang, R. Hesketh, L. Yang, and C. Zheng (2020-02) Time course of lung changes on chest ct during recovery from 2019 novel coronavirus (covid-19) pneumonia. Radiology 295, pp. 200370. External Links: Document Cited by: §1.
  • [21] Paul Mooney Chest X-Ray Images (Pneumonia). External Links: Link Cited by: §1.
  • [22] Radiological Society of North America RSNA Challenge. External Links: Link Cited by: §1.
  • [23] P. Rajpurkar, J. Irvin, K. Zhu, B. Yang, H. Mehta, T. Duan, D. Ding, A. Bagul, C. Langlotz, K. Shpanskaya, M. P. Lungren, and A. Y. Ng (2017) Chexnet: radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225. Cited by: §1.
  • [24] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra (2017-10) Grad-cam: visual explanations from deep networks via gradient-based localization. In The IEEE International Conference on Computer Vision (ICCV), Cited by: §3.2.
  • [25] F. Shi, J. Wang, J. Shi, Z. Wu, Q. Wang, Z. Tang, K. He, Y. Shi, and D. Shen (2020) Review of artificial intelligence techniques in imaging data acquisition, segmentation and diagnosis for covid-19. IEEE Reviews in Biomedical Engineering. Cited by: §1.
  • [26] H. Shi, X. Han, N. Jiang, Y. Cao, O. Alwalid, J. Gu, Y. Fan, and C. Zheng (2020) Radiological findings from 81 patients with covid-19 pneumonia in wuhan, china: a descriptive study. The Lancet Infectious Diseases. Cited by: §1.
  • [27] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi (2017)

    Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

    .
    In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, pp. 4278–4284. Cited by: §2.4.
  • [28] M. Tan and Q. V. Le (2019) EfficientNet: rethinking model scaling for convolutional neural networks. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, K. Chaudhuri and R. Salakhutdinov (Eds.), Proceedings of Machine Learning Research, Vol. 97, pp. 6105–6114. External Links: Link Cited by: §2.4.
  • [29] D. Varshni, K. Thakral, L. Agarwal, R. Nijhawan, and A. Mittal (2019)

    Pneumonia detection using cnn based feature extraction

    .
    In 2019 IEEE International Conference on Electrical, Computer and Communication Technologies (ICECCT), Vol. , pp. 1–7. Cited by: §1.
  • [30] L. Wang and A. Wong (2020) COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Images. pp. 1–12. External Links: 2003.09871, Link Cited by: §1.
  • [31] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers (2017) Chestx-ray8: hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, pp. 3462–3471. Cited by: §1, §1, §4.
  • [32] Y. Wang, C. Dong, Y. Hu, C. Li, Q. Ren, X. Zhang, H. Shi, and M. Zhou (2020-03)

    Temporal changes of ct findings in 90 patients with covid-19 pneumonia: a longitudinal study

    .
    Radiology, pp. 200843. External Links: Document Cited by: §1.
  • [33] H. Y. F. Wong, H. Y. S. Lam, A. H. Fong, S. T. Leung, T. W. Chin, C. S. Y. Lo, M. M. Lui, J. C. Y. Lee, K. W. Chiu, T. Chung, et al. (2020) Frequency and distribution of chest radiographic findings in covid-19 positive patients. Radiology, pp. 201160. Cited by: §1.

Supplementary Material

Label EfficientNetBa4 InceptionResnetV2 XCeption
normal [C0205307] [8412] 0.86 0.83 0.86
exclude [197] 0.55 0.46 0.49
suboptimal study [C2828075] [201] 0.80 0.78 0.77
radiological finding [C0436485] [12877] 0.85 0.81 0.84
├── unchanged [1713] 0.70 0.67 0.68
├── obesity [C0028754] [5] 0.84 0.68 0.65
├── chronic changes [C0742362] [978] 0.79 0.76 0.78
├── calcified densities [C2203586] [199] 0.71 0.68 0.70
├── calcified granuloma [C0333404] [458] 0.82 0.73 0.80
├── calcified adenopathy [83] 0.72 0.69 0.73
└── calcified mediastinal adenopathy [3] 0.56 0.68 0.68
├── calcified pleural thickening [88] 0.93 0.89 0.93
├── calcified pleural plaques [34] 0.91 0.87 0.91
├── heart valve calcified [C2073448] [23] 0.88 0.88 0.87
└── calcified fibroadenoma [10] 0.80 0.63 0.66
├── granuloma [C0235557] [557] 0.82 0.72 0.79
└── calcified granuloma [C0333404] [458] 0.82 0.73 0.80
├── end on vessel [37] 0.58 0.64 0.68
├── adenopathy [C0478664] [131] 0.69 0.66 0.68
└── calcified adenopathy [83] 0.72 0.69 0.73
├── nodule [C0034079] [576] 0.75 0.70 0.73
└── multiple nodules [C2073563] [45] 0.92 0.86 0.88
├── pseudonodule [438] 0.73 0.68 0.73
├── nipple shadow [231] 0.83 0.72 0.79
└── end on vessel [37] 0.58 0.64 0.68
├── abscess [C0024110] [1] 0.98 0.78 0.91
├── cyst [C2073485] [1] 0.81 0.96 0.96
├── cavitation [C0578537] [77] 0.94 0.94 0.95
├── fibrotic band [C0865843] [538] 0.86 0.80 0.84
├── volume loss [C3203358] [363] 0.92 0.89 0.92
├── hypoexpansion [144] 0.83 0.78 0.76
└── hypoexpansion basal [19] 0.84 0.72 0.74
├── bullas [C0241982] [92] 0.87 0.86 0.85
├── pneumothorax [C2073565] [59] 0.77 0.71 0.75
└── hydropneumothorax [C0020303] [5] 0.93 0.85 0.90
├── pneumoperitoneo [C0032320] [7] 0.79 0.64 0.80
├── pneumomediastinum [C2073636] [2] 0.44 0.25 0.58
├── subcutaneous emphysema [C0038536] [8] 0.93 0.91 0.94
├── hyperinflated lung [C0546312] [94] 0.80 0.75 0.73
├── flattened diaphragm [C2073504] [136] 0.74 0.70 0.72
├── lung vascular paucity [31] 0.73 0.72 0.72
├── air trapping [C0231819] [798] 0.75 0.71 0.72
├── bronchiectasis [C0006267] [363] 0.81 0.78 0.80
├── infiltrates [C0277877] [2103] 0.87 0.84 0.86
├── interstitial pattern [C2073538] [907] 0.85 0.81 0.82
├── ground glass pattern [C3544344] [53] 0.84 0.88 0.84
├── reticular interstitial pattern [96] 0.82 0.74 0.74
├── reticulonodular interstitial pattern [C2073672] [38] 0.86 0.83 0.82
└── miliary opacities [C2073583] [25] 0.72 0.73 0.72
└── alveolar pattern [C1332240] [683] 0.92 0.91 0.92
├── consolidation [C0521530] [315] 0.92 0.90 0.92
└── air bronchogram [C3669021] [16] 0.77 0.76 0.70
└── air bronchogram [C3669021] [16] 0.77 0.76 0.70
├── bronchovascular markings [C2073518] [182] 0.82 0.75 0.80
├── air fluid level [C0740844] [24] 0.75 0.75 0.71
├── increased density [C1443940] [540] 0.80 0.78 0.78
├── atelectasis [C0004144] [1002] 0.85 0.74 0.81
├── total atelectasis [C0264497] [2] 0.99 0.94 0.85
├── lobar atelectasis [88] 0.89 0.83 0.87
├── segmental atelectasis [21] 0.73 0.77 0.72
├── laminar atelectasis [698] 0.84 0.70 0.78
├── round atelectasis [C2062952] [1] 0.93 0.91 0.94
└── atelectasis basal [C0746053] [3] 0.95 0.78 0.86
├── mediastinal shift [C0264576] [9] 0.73 0.75 0.69
├── azygos lobe [C0265794] [74] 0.56 0.61 0.62
├── fissure thickening [80] 0.89 0.78 0.84
├── minor fissure thickening [61] 0.90 0.77 0.84
├── major fissure thickening [10] 0.78 0.68 0.76
└── loculated fissural effusion [20] 0.91 0.86 0.89
├── pleural thickening [C0264545] [789] 0.85 0.80 0.83
├── apical pleural thickening [585] 0.86 0.80 0.84
└── calcified pleural thickening [88] 0.93 0.89 0.93
├── pleural plaques [C0340030] [38] 0.88 0.85 0.88
└── calcified pleural plaques [34] 0.91 0.87 0.91
├── pleural effusion [C2073625] [786] 0.96 0.91 0.95
├── loculated pleural effusion [C0747639] [29] 0.95 0.92 0.96
├── loculated fissural effusion [20] 0.91 0.86 0.89
├── hydropneumothorax [C0020303] [5] 0.93 0.85 0.90
├── empyema [C0014009] [0] 0.00 0.00 0.00
└── hemothorax [C0019123] [0] 0.00 0.00 0.00
├── pleural mass [C1709576] [0] 0.00 0.00 0.00
├── costophrenic angle blunting [C0742855] [810] 0.89 0.78 0.85
├── vascular redistribution [C0239041] [64] 0.88 0.85 0.87
└── central vascular redistribution [14] 0.83 0.83 0.88
├── hilar enlargement [C1698506] [868] 0.73 0.68 0.71
├── adenopathy [C0149711] [131] 0.69 0.66 0.68
└── vascular hilar enlargement [602] 0.76 0.68 0.73
└── pulmonary artery enlargement [C2072932] [36] 0.69 0.64 0.63
├── hilar congestion [C0582411] [80] 0.88 0.89 0.88
├── cardiomegaly [C0018800] [2068] 0.93 0.91 0.92
├── pericardial effusion [C0031039] [11] 0.90 0.92 0.79
├── kerley lines [C0239019] [18] 0.81 0.79 0.80
├── dextrocardia [C0011813] [1] 0.86 0.98 0.96
├── right sided aortic arch [C0035615] [7] 0.74 0.57 0.71
├── aortic atheromatosis [C1096249] [364] 0.83 0.81 0.81
├── aortic elongation [1728] 0.90 0.87 0.89
├── descendent aortic elongation [C4476542] [174] 0.87 0.82 0.84
├── ascendent aortic elongation [C3889085] [31] 0.79 0.78 0.74
├── aortic button enlargement [C1851119] [76] 0.78 0.75 0.78
└── supra aortic elongation [238] 0.87 0.84 0.86
├── aortic aneurysm [C0003486] [7] 0.94 0.74 0.89
├── mediastinal enlargement [C2021206] [646] 0.82 0.79 0.81
├── superior mediastinal enlargement [C4273001] [414] 0.82 0.78 0.81
├── goiter [C0018021] [175] 0.78 0.73 0.80
└── supra aortic elongation [238] 0.87 0.84 0.86
├── descendent aortic elongation [C4476542] [174] 0.87 0.82 0.84
├── ascendent aortic elongation [C3889085] [31] 0.79 0.78 0.74
├── aortic aneurysm [C0003486] [7] 0.94 0.74 0.89
├── mediastinal mass [C0240318] [41] 0.73 0.67 0.73
└── hiatal hernia [C3489393] [368] 0.87 0.80 0.85
├── tracheal shift [148] 0.76 0.66 0.79
├── mass [C2603353] [175] 0.80 0.76 0.79
├── mediastinal mass [C0240318] [41] 0.73 0.67 0.73
├── breast mass [C0024103] [0] 0.00 0.00 0.00
├── pleural mass [C1709576] [0] 0.00 0.00 0.00
├── pulmonary mass [C0149726] [117] 0.90 0.86 0.90
└── soft tissue mass [C0457196] [16] 0.57 0.63 0.51
├── esophagic dilatation [C0192389] [1] 0.90 0.85 0.86
├── azygoesophageal recess shift [7] 0.65 0.63 0.55
├── pericardial effusion [C0031039] [11] 0.90 0.92 0.79
├── lipomatosis [76] 0.71 0.70 0.64
└── mediastinic lipomatosis [C1333298] [76] 0.71 0.69 0.65
├── thoracic cage deformation [C4538889] [1847] 0.80 0.67 0.73
├── scoliosis [C0036439] [1264] 0.82 0.67 0.74
├── kyphosis [C2115817] [602] 0.81 0.77 0.78
├── pectum excavatum [C2051831] [47] 0.77 0.72 0.69
├── pectum carinatum [C2939416] [3] 0.58 0.47 0.63
└── cervical rib [C0158779] [16] 0.75 0.62 0.76
├── vertebral degenerative changes [C4290224] [934] 0.74 0.72 0.72
└── vertebral compression [C0262431] [290] 0.76 0.72 0.74
└── vertebral anterior compression [253] 0.75 0.72 0.75
├── lytic bone lesion [C0476382] [12] 0.73 0.75 0.71
├── sclerotic bone lesion [C4315325] [93] 0.65 0.58 0.62
└── blastic bone lesion [C2203581] [9] 0.83 0.74 0.73
├── costochondral junction hypertrophy [17] 0.62 0.57 0.64
├── sternoclavicular junction hypertrophy [1] 0.31 0.16 0.09
├── axial hyperostosis [C1400000] [34] 0.71 0.70 0.66
├── osteopenia [C0029453] [81] 0.86 0.80 0.80
├── osteoporosis [C0029456] [47] 0.88 0.83 0.88
├── non axial articular degenerative changes [34] 0.84 0.81 0.81
├── subacromial space narrowing [11] 0.67 0.76 0.68
├── fracture [C0016658] [591] 0.75 0.69 0.73
├── clavicle fracture [C0159658] [38] 0.57 0.56 0.55
├── humeral fracture [C0020162] [29] 0.79 0.74 0.73
├── vertebral fracture [C0080179] [36] 0.86 0.84 0.86
└── rib fracture [C0035522] [496] 0.75 0.69 0.73
└── callus rib fracture [C0006767] [414] 0.76 0.71 0.74
├── gynecomastia [C0018418] [107] 0.82 0.72 0.79
├── hiatal hernia [C3489393] [368] 0.87 0.80 0.85
├── Chilaiditi sign [C3178780] [1] 0.93 0.91 0.89
├── hemidiaphragm elevation [C2073707] [300] 0.83 0.72 0.77
├── diaphragmatic eventration [C0011981] [182] 0.78 0.72 0.72
├── tracheostomy tube [C0184159] [40] 0.98 0.85 0.96
├── endotracheal tube [C0336630] [16] 0.99 0.92 0.95
├── NSG tube [49] 0.91 0.82 0.86
├── chest drain tube [C0008034] [39] 0.91 0.87 0.90
├── ventriculoperitoneal drain tube [C0162702] [5] 0.49 0.55 0.46
├── gastrostomy tube [C0150595] [2] 0.91 0.59 0.96
├── nephrostomy tube [C0184149] [0] 0.00 0.00 0.00
├── double J stent [C0441293] [2] 0.51 0.51 0.68
├── catheter [C0085590] [137] 0.92 0.86 0.92
└── central venous catheter [C1145640] [125] 0.94 0.89 0.93
├── cvc via subclavian vein [C0398281] [22] 0.92 0.85 0.91
├── cvc via jugular vein [C0398278] [18] 0.91 0.93 0.88
├── reservoir cvc [C2026143] [66] 0.96 0.91 0.97
└── cvc via umbilical vein [C0398284] [0] 0.00 0.00 0.00
├── electrical device [406] 0.99 0.99 0.99
├── dual chamber device [C2732817] [164] 0.99 0.99 0.99
├── single chamber device [C2733312] [69] 0.99 0.99 0.99
├── pacemaker [C0030163] [367] 1.00 1.00 1.00
└── dai [C0972395] [36] 0.98 0.96 0.96
├── artificial heart valve [C1399223] [146] 0.99 0.98 0.99
├── artificial mitral heart valve [C0869752] [56] 0.99 0.98 0.99
└── artificial aortic heart valve [C0869748] [46] 0.99 0.97 0.99
├── surgery [995] 0.89 0.78 0.85
├── metal [C0025552] [690] 0.92 0.85 0.91
├── osteosynthesis material [C0016642] [101] 0.86 0.62 0.82
├── sternotomy [C0185792] [382] 1.00 0.97 1.00
└── suture material [C4305366] [267] 0.90 0.85 0.88
├── bone cement [C0005934] [0] 0.00 0.00 0.00
├── prosthesis [C0175649] [117] 0.84 0.66 0.77
├── humeral prosthesis [C0745058] [12] 0.97 0.75 0.94
├── mammary prosthesis [C0917968] [95] 0.85 0.70 0.77
└── endoprosthesis [C0005846] [7] 0.79 0.77 0.76
└── aortic endoprosthesis [2] 0.95 0.92 0.99
├── surgery breast [C3714726] [175] 0.83 0.70 0.75
└── mastectomy [C0024881] [109] 0.81 0.66 0.75
├── surgery neck [C0185773] [9] 0.70 0.67 0.77
├── surgery lung [C0038903] [44] 0.82 0.83 0.81
├── surgery heart [C0018821] [2] 0.99 0.94 0.91
└── surgery humeral [C0186326] [2] 0.99 0.86 0.93
├── abnormal foreign body [C0016542] [4] 0.94 0.84 0.94
└── external foreign body [11] 0.51 0.65 0.59
differential diagnosis [4852] 0.82 0.79 0.81
├── pneumonia [C0032285] [1189] 0.88 0.86 0.87
└── atypical pneumonia [C1412002] [340] 0.93 0.92 0.91
└── viral pneumonia [C0032310] [318] 0.94 0.93 0.92
├── COVID 19 [C5203670] [307] 0.94 0.93 0.92
└── COVID 19 uncertain [C5203671] [6] 0.84 0.85 0.82
├── tuberculosis [C0041296] [189] 0.93 0.88 0.92
└── tuberculosis sequelae [C0494132] [148] 0.95 0.91 0.94
├── lung metastasis [C0153676] [49] 0.91 0.87 0.86
├── lymphangitis carcinomatosa [C0238258] [5] 0.97 0.96 0.98
├── lepidic adenocarcinoma [C4049711] [1] 0.91 0.97 0.87
├── pulmonary fibrosis [C0034069] [226] 0.92 0.86 0.89
├── post radiotherapy changes [C1320687] [44] 0.94 0.75 0.94
└── asbestosis signs [C0003949] [11] 0.96 0.87 0.94
├── emphysema [C0034067] [208] 0.87 0.87 0.85
├── COPD signs [C0024117] [3129] 0.83 0.80 0.82
├── heart insufficiency [C0018801] [163] 0.94 0.93 0.92
├── respiratory distress [C0476273] [0] 0.00 0.00 0.00
├── pulmonary hypertension [C0020542] [24] 0.90 0.87 0.91
├── pulmonary artery hypertension [C2973725] [2] 0.94 0.74 0.92
└── pulmonary venous hypertension [C4477098] [1] 0.91 0.87 0.93
├── pulmonary edema [C0034063] [53] 0.96 0.96 0.96
└── bone metastasis [C0153690] [23] 0.86 0.80 0.83
localization [12138] 0.83 0.80 0.82
├── extracorporal [C0424529] [0] 0.00 0.00 0.00
├── cervical [C0920882] [87] 0.66 0.62 0.69
├── soft tissue [C0225317] [726] 0.77 0.66 0.71
├── subcutaneous [C0443315] [23] 0.94 0.85 0.92
├── axilar [C0004454] [172] 0.80 0.68 0.73
└── pectoral [C0230111] [468] 0.81 0.67 0.76
└── nipple [C0028109] [224] 0.82 0.70 0.79
├── bone [C0262950] [1727] 0.70 0.66 0.67
├── shoulder [C0037004] [242] 0.80 0.68 0.76
├── acromioclavicular joint [C0001208] [16] 0.70 0.57 0.62
├── rotator cuff [C0085515] [17] 0.59 0.58 0.61
├── supraspisnous [C0225001] [9] 0.70 0.73 0.62
└── humerus [C0020164] [172] 0.80 0.68 0.77
├── humeral head [C0223683] [62] 0.73 0.63 0.73
├── humeral neck [C0448034] [13] 0.79 0.71 0.69
└── glenohumeral joint [C0225063] [14] 0.86 0.76 0.81
├── clavicle [C0008913] [65] 0.59 0.52 0.57
├── scapula [C0036277] [43] 0.65 0.53 0.64
├── costoesternal [C0450216] [9] 0.60 0.63 0.61
├── column [C0037949] [599] 0.71 0.70 0.69
├── intersomatic space [C0223088] [15] 0.64 0.63 0.67
├── dorsal vertebrae [C0039987] [428] 0.71 0.70 0.69
├── cervical vertebrae [C3665420] [19] 0.67 0.55 0.67
├── lumbar vertebrae [C0024091] [28] 0.77 0.77 0.76
└── paravertebral [C0442150] [41] 0.65 0.66 0.61
└── rib [C0035561] [760] 0.72 0.66 0.68
├── anterior rib [C4323264] [73] 0.66 0.60 0.61
├── posterior rib [C4323265] [134] 0.68 0.67 0.69
└── rib cartilage [C0222787] [7] 0.75 0.77 0.74
├── hemithorax [C0934569] [749] 0.84 0.81 0.83
├── extrapleural [C0442091] [21] 0.78 0.71 0.75
├── extrapulmonary [9] 0.68 0.64 0.64
├── pleural [C0032225] [2156] 0.84 0.80 0.84
├── subpleural [C0225775] [40] 0.89 0.84 0.86
├── fissure [C0458078] [147] 0.85 0.75 0.82
├── minor fissure [C0734040] [83] 0.91 0.77 0.86
└── major fissure [C4253583] [28] 0.81 0.75 0.79
├── lobar [C0225752] [604] 0.79 0.76 0.78
├── upper lobe [C0225756] [1838] 0.86 0.78 0.84
├── left upper lobe [C1261076] [298] 0.83 0.76 0.81
└── lingula [C0225740] [214] 0.82 0.69 0.77
└── right upper lobe [C1261074] [568] 0.84 0.76 0.83
├── lower lobe [C0225758] [774] 0.82 0.77 0.81
├── left lower lobe [C1261077] [348] 0.81 0.75 0.80
└── right lower lobe [C1261075] [228] 0.81 0.75 0.79
└── middle lobe [C4281590] [328] 0.80 0.74 0.77
├── subsegmental [C0929165] [351] 0.81 0.69 0.75
├── bronchi [C0006255] [621] 0.77 0.75 0.76
├── peribronchi [C0225607] [24] 0.85 0.76 0.82
├── lung field [C0225759] [7703] 0.83 0.78 0.82
├── pleural [C0032225] [2156] 0.84 0.80 0.84
├── subpleural [C0225775] [40] 0.89 0.84 0.86
├── major fissure [C4253583] [28] 0.81 0.75 0.79
├── subsegmental [C0929165] [351] 0.81 0.69 0.75
├── bronchi [C0006255] [621] 0.77 0.75 0.76
├── peribronchi [C0225607] [24] 0.85 0.76 0.82
├── upper lung field [C0929227] [1944] 0.86 0.78 0.84
└── upper lobe [C0225756] [1838] 0.86 0.78 0.84
├── left upper lobe [C1261076] [298] 0.83 0.76 0.81
├── right upper lobe [C1261074] [568] 0.84 0.76 0.83
├── apical [C0734296] [872] 0.86 0.80 0.84
└── suprahilar [C0934260] [49] 0.64 0.60 0.63
├── middle lung field [C0929434] [1960] 0.76 0.73 0.75
├── aortopulmonary window [C1282038] [32] 0.61 0.62 0.60
├── hilar [C0205150] [1533] 0.75 0.71 0.73
├── pulmonary artery [C0034052] [38] 0.69 0.64 0.65
├── hilar bilateral [86] 0.82 0.76 0.79
└── perihilar [C0225702] [335] 0.85 0.81 0.82
└── minor fissure [C0734040] [83] 0.91 0.77 0.86
└── lower lung field [4559] 0.81 0.76 0.80
├── basal [C1282378] [1625] 0.80 0.75 0.78
├── lower lobe [C0225758] [774] 0.82 0.77 0.81
├── left lower lobe [C1261077] [348] 0.81 0.75 0.80
└── right lower lobe [C1261075] [228] 0.81 0.75 0.79
├── middle lobe [C4281590] [328] 0.80 0.74 0.77
├── infrahilar [108] 0.77 0.70 0.77
├── lingula [C0225740] [214] 0.82 0.69 0.77
├── supradiaphragm [2] 0.84 0.48 0.69
├── diaphragm [C0011980] [721] 0.76 0.69 0.73
├── infradiaphragm [25] 0.84 0.77 0.81
├── cardiophrenic angle [67] 0.66 0.66 0.64
└── costophrenic angle [C0230151] [1235] 0.79 0.72 0.77
├── right costophrenic angle [C0504099] [285] 0.85 0.77 0.81
├── left costophrenic angle [C0504100] [359] 0.86 0.71 0.82
└── bilateral costophrenic angle [10] 0.87 0.85 0.91
├── peripheral [C0205100] [241] 0.95 0.93 0.93
├── central [C0205099] [111] 0.84 0.83 0.83
├── mediastinum [C0025066] [5136] 0.80 0.78 0.79
├── superior mediastinum [C0230147] [538] 0.84 0.80 0.83
├── carotid artery [C0007272] [1] 1.00 0.55 1.00
├── brachiocephalic veins [C0006095] [7] 0.86 0.82 0.88
├── supra aortic [228] 0.87 0.84 0.85
├── aortic button [C0003489] [80] 0.78 0.75 0.76
├── superior cave vein [C3165182] [77] 0.94 0.90 0.94
└── subclavian vein [C0038532] [117] 0.96 0.94 0.96
├── lower mediastinum [C1261193] [3489] 0.78 0.76 0.77
├── anterior mediastinum [C0230148] [17] 0.63 0.69 0.66
└── thymus [C0040113] [7] 0.61 0.74 0.53
├── middle mediastinum [C0230149] [3142] 0.80 0.78 0.79
└── cardiac [C1522601] [3134] 0.80 0.78 0.79
└── coronary [C1522318] [12] 0.82 0.79 0.82
└── posterior mediastinum [C0230150] [363] 0.73 0.69 0.71
└── retrocardiac [361] 0.73 0.69 0.72
└── aortic [C0003483] [2059] 0.88 0.85 0.87
├── paratracheal [C0442143] [91] 0.65 0.64 0.68
├── airways [C0458827] [900] 0.73 0.71 0.72
├── tracheal [C0040578] [294] 0.79 0.66 0.80
└── bronchi [C0006255] [621] 0.77 0.75 0.76
├── esophageal [C1522619] [24] 0.68 0.66 0.65
├── paramediastinum [19] 0.80 0.79 0.81
├── paracardiac [40] 0.76 0.69 0.72
├── epigastric [10] 0.69 0.62 0.60
├── gastric chamber [C3714551] [20] 0.81 0.74 0.71
├── hypochondrium [C0230186] [40] 0.66 0.68 0.71
├── right hypochondrium [C0738590] [25] 0.63 0.62 0.69
└── gallbladder [C0016976] [13] 0.63 0.63 0.67
└── left hypochondrium [C0738591] [15] 0.68 0.73 0.76
├── right [C0444532] [3594] 0.76 0.72 0.74
├── left [C0443246] [2877] 0.77 0.73 0.76
└── bilateral [C0238767] [1913] 0.82 0.80 0.80
├── diffuse bilateral [149] 0.91 0.86 0.87
└── basal bilateral [435] 0.82 0.77 0.79