DeepAI AI Chat
Log In Sign Up

CORAL8: Concurrent Object Regression for Area Localization in Medical Image Panels

by   Sam Maksoud, et al.

This work tackles the problem of generating a medical report for multi-image panels. We apply our solution to the Renal Direct Immunofluorescence (RDIF) assay which requires a pathologist to generate a report based on observations across the eight different WSI in concert with existing clinical features. To this end, we propose a novel attention-based multi-modal generative recurrent neural network (RNN) architecture capable of dynamically sampling image data concurrently across the RDIF panel. The proposed methodology incorporates text from the clinical notes of the requesting physician to regulate the output of the network to align with the overall clinical context. In addition, we found the importance of regularizing the attention weights for word generation processes. This is because the system can ignore the attention mechanism by assigning equal weights for all members. Thus, we propose two regularizations which force the system to utilize the attention mechanism. Experiments on our novel collection of RDIF WSIs provided by a large clinical laboratory demonstrate that our framework offers significant improvements over existing methods.


page 1

page 2

page 3

page 4


Heterogeneous Graph Learning for Multi-modal Medical Data Analysis

Routine clinical visits of a patient produce not only image data, but al...

Explainable Prediction of Adverse Outcomes Using Clinical Notes

Clinical notes contain a large amount of clinically valuable information...

Transformer-based Personalized Attention Mechanism (PersAM) for Medical Images with Clinical Records

In medical image diagnosis, identifying the attention region, i.e., the ...

Explainable Prediction of Medical Codes from Clinical Text

Clinical notes are text documents that are created by clinicians for eac...

To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regression

Given an untrimmed video and a sentence description, temporal sentence l...

R2D2: Relational Text Decoding with Transformers

We propose a novel framework for modeling the interaction between graphi...