Improving Pneumonia Localization via Cross-Attention on Medical Images and Reports

by   Riddhish Bhalodia, et al.

Localization and characterization of diseases like pneumonia are primary steps in a clinical pipeline, facilitating detailed clinical diagnosis and subsequent treatment planning. Additionally, such location annotated datasets can provide a pathway for deep learning models to be used for downstream tasks. However, acquiring quality annotations is expensive on human resources and usually requires domain expertise. On the other hand, medical reports contain a plethora of information both about pneumonia characteristics and its location. In this paper, we propose a novel weakly-supervised attention-driven deep learning model that leverages encoded information in medical reports during training to facilitate better localization. Our model also performs classification of attributes that are associated to pneumonia and extracted from medical reports for supervision. Both the classification and localization are trained in conjunction and once trained, the model can be utilized for both the localization and characterization of pneumonia using only the input image. In this paper, we explore and analyze the model using chest X-ray datasets and demonstrate qualitatively and quantitatively that the introduction of textual information improves pneumonia localization. We showcase quantitative results on two datasets, MIMIC-CXR and Chest X-ray-8, and we also showcase severity characterization on the COVID-19 dataset.


page 4

page 5

page 8


Weakly-Supervised Segmentation for Disease Localization in Chest X-Ray Images

Deep Convolutional Neural Networks have proven effective in solving the ...

Automated Enriched Medical Concept Generation for Chest X-ray Images

Decision support tools that rely on supervised learning require large am...

Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignment

Self-supervised learning provides an opportunity to explore unlabeled ch...

Thoracic Disease Identification and Localization with Limited Supervision

Accurate identification and localization of abnormalities from radiology...

Learning Hierarchical Attention for Weakly-supervised Chest X-Ray Abnormality Localization and Diagnosis

We consider the problem of abnormality localization for clinical applica...

Deep Learning to Quantify Pulmonary Edema in Chest Radiographs

Background: Clinical management decisions for acutely decompensated CHF ...

Automated Radiological Report Generation For Chest X-Rays With Weakly-Supervised End-to-End Deep Learning

The chest X-Ray (CXR) is the one of the most common clinical exam used t...

Please sign up or login with your details

Forgot password? Click here to reset