DeepAI AI Chat
Log In Sign Up

Improving Pneumonia Localization via Cross-Attention on Medical Images and Reports

10/06/2021
by   Riddhish Bhalodia, et al.
THE UNIVERSITY OF UTAH
National Institutes of Health
Nvidia
0

Localization and characterization of diseases like pneumonia are primary steps in a clinical pipeline, facilitating detailed clinical diagnosis and subsequent treatment planning. Additionally, such location annotated datasets can provide a pathway for deep learning models to be used for downstream tasks. However, acquiring quality annotations is expensive on human resources and usually requires domain expertise. On the other hand, medical reports contain a plethora of information both about pneumonia characteristics and its location. In this paper, we propose a novel weakly-supervised attention-driven deep learning model that leverages encoded information in medical reports during training to facilitate better localization. Our model also performs classification of attributes that are associated to pneumonia and extracted from medical reports for supervision. Both the classification and localization are trained in conjunction and once trained, the model can be utilized for both the localization and characterization of pneumonia using only the input image. In this paper, we explore and analyze the model using chest X-ray datasets and demonstrate qualitatively and quantitatively that the introduction of textual information improves pneumonia localization. We showcase quantitative results on two datasets, MIMIC-CXR and Chest X-ray-8, and we also showcase severity characterization on the COVID-19 dataset.

READ FULL TEXT

page 4

page 5

page 8

07/01/2020

Weakly-Supervised Segmentation for Disease Localization in Chest X-Ray Images

Deep Convolutional Neural Networks have proven effective in solving the ...
10/07/2019

Automated Enriched Medical Concept Generation for Chest X-ray Images

Decision support tools that rely on supervised learning require large am...
09/04/2021

Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignment

Self-supervised learning provides an opportunity to explore unlabeled ch...
12/23/2021

Learning Hierarchical Attention for Weakly-supervised Chest X-Ray Abnormality Localization and Diagnosis

We consider the problem of abnormality localization for clinical applica...
11/17/2017

Thoracic Disease Identification and Localization with Limited Supervision

Accurate identification and localization of abnormalities from radiology...
08/13/2020

Deep Learning to Quantify Pulmonary Edema in Chest Radiographs

Background: Clinical management decisions for acutely decompensated CHF ...
02/27/2019

Semi-supervised Learning for Quantification of Pulmonary Edema in Chest X-Ray Images

We propose and demonstrate machine learning algorithms to assess the sev...