Gaze-Guided Class Activation Mapping: Leveraging Human Attention for Network Attention in Chest X-rays Classification

by   Hongzhi Zhu, et al.

The increased availability and accuracy of eye-gaze tracking technology has sparked attention-related research in psychology, neuroscience, and, more recently, computer vision and artificial intelligence. The attention mechanism in artificial neural networks is known to improve learning tasks. However, no previous research has combined the network attention and human attention. This paper describes a gaze-guided class activation mapping (GG-CAM) method to directly regulate the formation of network attention based on expert radiologists' visual attention for the chest X-ray pathology classification problem, which remains challenging due to the complex and often nuanced differences among images. GG-CAM is a lightweight (3 additional trainable parameters for regulating the learning process) and generic extension that can be easily applied to most classification convolutional neural networks (CNN). GG-CAM-modified CNNs do not require human attention as an input when fully trained. Comparative experiments suggest that two standard CNNs with the GG-CAM extension achieve significantly greater classification performance. The median area under the curve (AUC) metrics for ResNet50 increases from 0.721 to 0.776. For EfficientNetv2 (s), the median AUC increases from 0.723 to 0.801. The GG-CAM also brings better interpretability of the network that facilitates the weakly-supervised pathology localization and analysis.


Attention-based Saliency Maps Improve Interpretability of Pneumothorax Classification

Purpose: To investigate chest radiograph (CXR) classification performanc...

Creation and Validation of a Chest X-Ray Dataset with Eye-tracking and Report Dictation for AI Development

We developed a rich dataset of Chest X-Ray (CXR) images to assist invest...

Attention Based Glaucoma Detection: A Large-scale Database and CNN Model

Recently, the attention mechanism has been successfully applied in convo...

Matching Representations of Explainable Artificial Intelligence and Eye Gaze for Human-Machine Interaction

Rapid non-verbal communication of task-based stimuli is a challenge in h...

Human Attention in Fine-grained Classification

The way humans attend to, process and classify a given image has the pot...

Eye-gaze-guided Vision Transformer for Rectifying Shortcut Learning

Learning harmful shortcuts such as spurious correlations and biases prev...

An Integrated System for 3D Gaze Recovery and Semantic Analysis of Human Attention

This work describes a computer vision system that enables pervasive mapp...

Please sign up or login with your details

Forgot password? Click here to reset