ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models

by   Sheng Wang, et al.

Large language models (LLMs) have recently demonstrated their potential in clinical applications, providing valuable medical knowledge and advice. For example, a large dialog LLM like ChatGPT has successfully passed part of the US medical licensing exam. However, LLMs currently have difficulty processing images, making it challenging to interpret information from medical images, which are rich in information that supports clinical decisions. On the other hand, computer-aided diagnosis (CAD) networks for medical images have seen significant success in the medical field by using advanced deep-learning algorithms to support clinical decision-making. This paper presents a method for integrating LLMs into medical-image CAD networks. The proposed framework uses LLMs to enhance the output of multiple CAD networks, such as diagnosis networks, lesion segmentation networks, and report generation networks, by summarizing and reorganizing the information presented in natural language text format. The goal is to merge the strengths of LLMs' medical domain knowledge and logical reasoning with the vision understanding capability of existing medical-image CAD models to create a more user-friendly and understandable system for patients compared to conventional CAD systems. In the future, LLM's medical knowledge can be also used to improve the performance of vision-based medical-image CAD models.


page 4

page 7

page 10

page 11


Fusing Medical Image Features and Clinical Features with Deep Learning for Computer-Aided Diagnosis

Current Computer-Aided Diagnosis (CAD) methods mainly depend on medical ...

ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMs

The potential of integrating Computer-Assisted Diagnosis (CAD) with Larg...

Goldilocks-curriculum Domain Randomization and Fractal Perlin Noise with Application to Sim2Real Pneumonia Lesion Detection

A computer-aided detection (CAD) system based on machine learning is exp...

Unifying Neural Learning and Symbolic Reasoning for Spinal Medical Report Generation

Automated medical report generation in spine radiology, i.e., given spin...

Label Stability in Multiple Instance Learning

We address the problem of instance label stability in multiple instance ...

Diagnosis of liver disease using computer-assisted imaging techniques: A Review

The evidence says that liver disease detection using CAD is one of the m...

Follow My Eye: Using Gaze to Supervise Computer-Aided Diagnosis

When deep neural network (DNN) was first introduced to the medical image...

Please sign up or login with your details

Forgot password? Click here to reset