Medical Image Captioning via Generative Pretrained Transformers

09/28/2022
by   Alexander Selivanov, et al.
0

The automatic clinical caption generation problem is referred to as proposed model combining the analysis of frontal chest X-Ray scans with structured patient information from the radiology records. We combine two language models, the Show-Attend-Tell and the GPT-3, to generate comprehensive and descriptive radiology records. The proposed combination of these models generates a textual summary with the essential information about pathologies found, their location, and the 2D heatmaps localizing each pathology on the original X-Ray scans. The proposed model is tested on two medical datasets, the Open-I, MIMIC-CXR, and the general-purpose MS-COCO. The results measured with the natural language assessment metrics prove their efficient applicability to the chest X-Ray image captioning.

READ FULL TEXT

page 1

page 5

page 6

page 10

research
04/26/2021

Contextualized Keyword Representations for Multi-modal Retinal Image Captioning

Medical image captioning automatically generates a medical description t...
research
02/11/2022

ACORT: A Compact Object Relation Transformer for Parameter Efficient Image Captioning

Recent research that applies Transformer-based architectures to image ca...
research
06/12/2019

Vispi: Automatic Visual Perception and Interpretation of Chest X-rays

Medical imaging contains the essential information for rendering diagnos...
research
04/04/2019

Clinically Accurate Chest X-Ray Report Generation

The automatic generation of radiology reports given medical radiographs ...
research
05/05/2022

Understanding Transfer Learning for Chest Radiograph Clinical Report Generation with Modified Transformer Architectures

The image captioning task is increasingly prevalent in artificial intell...
research
06/18/2023

Generation of Radiology Findings in Chest X-Ray by Leveraging Collaborative Knowledge

Among all the sub-sections in a typical radiology report, the Clinical I...

Please sign up or login with your details

Forgot password? Click here to reset