[Paper Review] Leveraging Human Selective Attention for Medical Image Analysis with Limited Training Data

Yifei Huang, Xiaoxiao Li+7 more|arXiv (Cornell University)|Dec 2, 2021

Advanced Neural Network Applications50 references4 citations

TL;DR

This paper proposes a novel framework that leverages human gaze data to enhance medical image analysis under limited training data by simulating selective attention via a Selective Attention Network (SAN) and an Auxiliary Attention Block (AAB). The AAB integrates gaze-derived attention maps into a backbone encoder using a modified multi-head attention mechanism, achieving state-of-the-art performance on 3D brain tumor segmentation and 2D chest X-ray classification with minimal labeled data.

ABSTRACT

The human gaze is a cost-efficient physiological data that reveals human underlying attentional patterns. The selective attention mechanism helps the cognition system focus on task-relevant visual clues by ignoring the presence of distractors. Thanks to this ability, human beings can efficiently learn from a very limited number of training samples. Inspired by this mechanism, we aim to leverage gaze for medical image analysis tasks with small training data. Our proposed framework includes a backbone encoder and a Selective Attention Network (SAN) that simulates the underlying attention. The SAN implicitly encodes information such as suspicious regions that is relevant to the medical diagnose tasks by estimating the actual human gaze. Then we design a novel Auxiliary Attention Block (AAB) to allow information from SAN to be utilized by the backbone encoder to focus on selective areas. Specifically, this block uses a modified version of a multi-head attention layer to simulate the human visual search procedure. Note that the SAN and AAB can be plugged into different backbones, and the framework can be used for multiple medical image analysis tasks when equipped with task-specific heads. Our method is demonstrated to achieve superior performance on both 3D tumor segmentation and 2D chest X-ray classification tasks. We also show that the estimated gaze probability map of the SAN is consistent with an actual gaze fixation map obtained by board-certified doctors.

Motivation & Objective

• To improve medical image analysis performance when training data is scarce, inspired by human selective attention mechanisms.
• To model human visual attention in medical imaging using gaze data collected from radiologists.
• To design a plug-and-play module that transfers gaze-based attention to backbone networks for segmentation and classification.
• To validate the framework on both 3D and 2D medical imaging tasks with real expert gaze data.
• To demonstrate that gaze-estimated attention maps align with actual human fixation patterns in clinical settings.

Proposed method

• A Selective Attention Network (SAN) is trained on actual human gaze data from board-certified radiologists to estimate task-relevant attention maps.
• An Auxiliary Attention Block (AAB) uses a modified multi-head attention mechanism to inject SAN-predicted gaze features into the backbone encoder as query vectors.
• The AAB uses backbone features as keys and values, while SAN outputs serve as queries, enabling spatially aware feature refinement.
• The framework is compatible with various backbone architectures (e.g., EfficientNet, ResNet, MobileNet) and task-specific heads for segmentation and classification.
• The method is trained end-to-end with cross-entropy loss for the main task and gaze estimation loss for the SAN.
• The gaze data is collected using eye-tracking during diagnostic screening tasks on BraTS 2020 and MIMIC-CXR-gaze datasets.

Experimental results

Research questions

• RQ1 Can human gaze data improve medical image analysis performance when labeled data is limited?
• RQ2 How well can a learned gaze estimation network (SAN) predict attention patterns consistent with actual radiologist fixations?
• RQ3 Does integrating gaze-derived attention via the AAB module improve feature learning in backbone networks for medical tasks?
• RQ4 How generalizable is the framework across different imaging modalities (2D X-ray, 3D MRI) and backbone architectures?
• RQ5 Can gaze data alone, without direct access to target data, enhance model generalization in zero-shot or few-shot settings?

Key findings

• The proposed framework achieves state-of-the-art performance on BraTS 2020 for 3D brain tumor segmentation using only 0.3 of the full training data, outperforming all baselines.
• On the MIMIC-CXR-gaze dataset, the method achieves an average AUROC of 59.26% with 0.7 of the training data, significantly outperforming the backbone (55.14%) and gaze-baseline (55.07%).
• The estimated gaze probability maps from the SAN show strong spatial consistency with actual fixation maps collected from board-certified radiologists.
• The AAB module enables performance gains across multiple backbones, with EfficientNet-b0 showing superior performance on limited data compared to ResNet-18 and MobileNet-v2.
• The framework demonstrates consistent improvement across all disease classes in the MIMIC-CXR-gaze dataset, especially for cardiomegaly (61.31% AUROC) and pleural effusion (61.94% AUROC) at 0.7 data ratio.
• The ablation study confirms that performance gains are due to the AAB design and not merely the inclusion of gaze data, as shown by the superior performance over both +gaze* and gaze loss baselines.