Detecting Spurious Correlations with Sanity Tests for Artificial Intelligence Guided Radiology Systems

03/04/2021
by   Usman Mahmood, et al.
14

Artificial intelligence (AI) has been successful at solving numerous problems in machine perception. In radiology, AI systems are rapidly evolving and show progress in guiding treatment decisions, diagnosing, localizing disease on medical images, and improving radiologists' efficiency. A critical component to deploying AI in radiology is to gain confidence in a developed system's efficacy and safety. The current gold standard approach is to conduct an analytical validation of performance on a generalization dataset from one or more institutions, followed by a clinical validation study of the system's efficacy during deployment. Clinical validation studies are time-consuming, and best practices dictate limited re-use of analytical validation data, so it is ideal to know ahead of time if a system is likely to fail analytical or clinical validation. In this paper, we describe a series of sanity tests to identify when a system performs well on development data for the wrong reasons. We illustrate the sanity tests' value by designing a deep learning system to classify pancreatic cancer seen in computed tomography scans.

READ FULL TEXT

page 1

page 4

page 7

page 10

research
09/02/2021

Artificial Intelligence in Dry Eye Disease

Dry eye disease (DED) has a prevalence of between 5 and 50%, depending o...
research
05/08/2019

A new direction to promote the implementation of artificial intelligence in natural clinical settings

Artificial intelligence (AI) researchers claim that they have made great...
research
06/04/2022

Future Artificial Intelligence tools and perspectives in medicine

Purpose of review: Artificial intelligence (AI) has become popular in me...
research
06/05/2023

AI Techniques for Cone Beam Computed Tomography in Dentistry: Trends and Practices

Cone-beam computed tomography (CBCT) is a popular imaging modality in de...
research
03/06/2023

Applying Artificial Intelligence to Clinical Decision Support in Mental Health: What Have We Learned?

Clinical decision support systems (CDSS) augmented with artificial intel...
research
04/14/2021

Towards a framework for evaluating the safety, acceptability and efficacy of AI systems for health: an initial synthesis

The potential presented by Artificial Intelligence (AI) for healthcare h...
research
09/25/2020

Democratizing Artificial Intelligence in Healthcare: A Study of Model Development Across Two Institutions Incorporating Transfer Learning

The training of deep learning models typically requires extensive data, ...

Please sign up or login with your details

Forgot password? Click here to reset