Evaluation of wait time saving effectiveness of triage algorithms

by   Yee Lam Elim Thompson, et al.

In the past decade, Artificial Intelligence (AI) algorithms have made promising impacts to transform healthcare in all aspects. One application is to triage patients' radiological medical images based on the algorithm's binary outputs. Such AI-based prioritization software is known as computer-aided triage and notification (CADt). Their main benefit is to speed up radiological review of images with time-sensitive findings. However, as CADt devices become more common in clinical workflows, there is still a lack of quantitative methods to evaluate a device's effectiveness in saving patients' waiting times. In this paper, we present a mathematical framework based on queueing theory to calculate the average waiting time per patient image before and after a CADt device is used. We study four workflow models with multiple radiologists (servers) and priority classes for a range of AI diagnostic performance, radiologist's reading rates, and patient image (customer) arrival rates. Due to model complexity, an approximation method known as the Recursive Dimensionality Reduction technique is applied. We define a performance metric to measure the device's time-saving effectiveness. A software tool is developed to simulate clinical workflow of image review/interpretation, to verify theoretical results, and to provide confidence intervals of the performance metric we defined. It is shown quantitatively that a triage device is more effective in a busy, short-staffed setting, which is consistent with our clinical intuition and simulation results. Although this work is motivated by the need for evaluating CADt devices, the framework we present in this paper can be applied to any algorithm that prioritizes customers based on its binary outputs.


Artificial Intelligence in Dry Eye Disease

Dry eye disease (DED) has a prevalence of between 5 and 50%, depending o...

Practical Statistical Considerations for the Clinical Validation of AI/ML-enabled Medical Diagnostic Devices

Artificial Intelligence (AI) and Machine-Learning (ML) models have been ...

Future Artificial Intelligence tools and perspectives in medicine

Purpose of review: Artificial intelligence (AI) has become popular in me...

Assistive Chatbots for healthcare: a succinct review

Artificial Intelligence (AI) for supporting healthcare services has neve...

Towards the Use of Saliency Maps for Explaining Low-Quality Electrocardiograms to End Users

When using medical images for diagnosis, either by clinicians or artific...

Development and Clinical Evaluation of an AI Support Tool for Improving Telemedicine Photo Quality

Telemedicine utilization was accelerated during the COVID-19 pandemic, a...

Blockchain-based Medical Image Sharing and Automated Critical-results Notification: A Novel Framework

In teleradiology, medical images are transmitted to offsite radiologists...

Please sign up or login with your details

Forgot password? Click here to reset