Medical-VLBERT: Medical Visual Language BERT for COVID-19 CT Report Generation With Alternate Learning

08/11/2021
by   Guangyi Liu, et al.
14

Medical imaging technologies, including computed tomography (CT) or chest X-Ray (CXR), are largely employed to facilitate the diagnosis of the COVID-19. Since manual report writing is usually too time-consuming, a more intelligent auxiliary medical system that could generate medical reports automatically and immediately is urgently needed. In this article, we propose to use the medical visual language BERT (Medical-VLBERT) model to identify the abnormality on the COVID-19 scans and generate the medical report automatically based on the detected lesion regions. To produce more accurate medical reports and minimize the visual-and-linguistic differences, this model adopts an alternate learning strategy with two procedures that are knowledge pretraining and transferring. To be more precise, the knowledge pretraining procedure is to memorize the knowledge from medical texts, while the transferring procedure is to utilize the acquired knowledge for professional medical sentences generations through observations of medical images. In practice, for automatic medical report generation on the COVID-19 cases, we constructed a dataset of 368 medical findings in Chinese and 1104 chest CT scans from The First Affiliated Hospital of Jinan University, Guangzhou, China, and The Fifth Affiliated Hospital of Sun Yat-sen University, Zhuhai, China. Besides, to alleviate the insufficiency of the COVID-19 training samples, our model was first trained on the large-scale Chinese CX-CHR dataset and then transferred to the COVID-19 CT dataset for further fine-tuning. The experimental results showed that Medical-VLBERT achieved state-of-the-art performances on terminology prediction and report generation with the Chinese COVID-19 CT dataset and the CX-CHR dataset. The Chinese COVID-19 CT dataset is available at https://covid19ct.github.io/.

READ FULL TEXT

page 1

page 4

page 7

page 9

page 10

page 12

research
05/13/2020

MosMedData: Chest CT Scans With COVID-19 Related Findings Dataset

This dataset contains anonymised human lung computed tomography (CT) sca...
research
06/01/2020

BIMCV COVID-19+: a large annotated dataset of RX and CT images from COVID-19 patients

In this work we describe BIMCV-COVID-19+ dataset, a large dataset from M...
research
11/22/2017

On the Automatic Generation of Medical Imaging Reports

Medical imaging is widely used in clinical practice for diagnosis and tr...
research
09/04/2023

An Empirical Analysis for Zero-Shot Multi-Label Classification on COVID-19 CT Scans and Uncurated Reports

The pandemic resulted in vast repositories of unstructured data, includi...
research
07/13/2021

A Dialogue-based Information Extraction System for Medical Insurance Assessment

In the Chinese medical insurance industry, the assessor's role is essent...
research
09/04/2022

Representative Image Feature Extraction via Contrastive Learning Pretraining for Chest X-ray Report Generation

Medical report generation is a challenging task since it is time-consumi...
research
03/26/2021

Mixing-AdaSIN: Constructing a de-biased dataset using Adaptive Structural Instance Normalization and texture Mixing

Following the pandemic outbreak, several works have proposed to diagnose...

Please sign up or login with your details

Forgot password? Click here to reset