[Paper Review] ACAT: Adversarial Counterfactual Attention for Classification and Detection in Medical Imaging

Alessandro Fontanella, Antreas Antoniou+5 more|arXiv (Cornell University)|Mar 27, 2023

COVID-19 diagnosis using AI4 citations

TL;DR

This paper proposes Adversarial Counterfactual Attention (ACAT), a novel framework that uses saliency maps derived from adversarially generated counterfactual images to create soft spatial attention masks at multiple scales, improving classification accuracy and robustness in medical imaging. ACAT boosts lesion classification accuracy on brain CT scans from 71.39% to 72.55% and on lung CT scans from 67.71% to 70.84%, outperforming baseline models and competing methods without requiring manual ROI annotations.

ABSTRACT

In some medical imaging tasks and other settings where only small parts of the image are informative for the classification task, traditional CNNs can sometimes struggle to generalise. Manually annotated Regions of Interest (ROI) are sometimes used to isolate the most informative parts of the image. However, these are expensive to collect and may vary significantly across annotators. To overcome these issues, we propose a framework that employs saliency maps to obtain soft spatial attention masks that modulate the image features at different scales. We refer to our method as Adversarial Counterfactual Attention (ACAT). ACAT increases the baseline classification accuracy of lesions in brain CT scans from 71.39% to 72.55% and of COVID-19 related findings in lung CT scans from 67.71% to 70.84% and exceeds the performance of competing methods. We investigate the best way to generate the saliency maps employed in our architecture and propose a way to obtain them from adversarially generated counterfactual images. They are able to isolate the area of interest in brain and lung CT scans without using any manual annotations. In the task of localising the lesion location out of 6 possible regions, they obtain a score of 65.05% on brain CT scans, improving the score of 61.29% obtained with the best competing method.

Motivation & Objective

• To address the challenge of low signal-to-noise ratios in medical imaging, where only small regions are diagnostically relevant, and traditional CNNs struggle to generalize.
• To eliminate reliance on expensive, inconsistent manual Region of Interest (ROI) annotations by learning attention masks from data-driven saliency maps.
• To improve model robustness and classification performance by modulating features with multi-scale attention masks derived from saliency maps.
• To investigate the effectiveness of different saliency map generation techniques, particularly adversarial counterfactuals, for isolating diagnostically relevant regions.
• To demonstrate that attention mechanisms based on saliency maps are more effective than random regularization or single-scale attention.

Proposed method

• ACAT employs a dual-branch architecture: one branch processes raw image features, and another processes saliency maps to generate soft spatial attention masks at multiple network scales.
• Saliency maps are generated using an adversarial counterfactual generation method that finds minimal latent space perturbations to shift an input toward a target class, as predicted by a pre-trained classifier.
• Attention masks are applied at three stages (early, middle, late) of the network to modulate feature maps, reducing noise from non-informative regions.
• An attention fusion layer combines the multi-scale attention masks using learnable weights to better inform the final classification decision.
• The framework is trained end-to-end, with the saliency map generator and classifier jointly optimized to improve feature modulation and prediction accuracy.
• Robustness is evaluated by measuring pre-activation variance in network layers, showing ACAT reduces variance and stabilizes training under input perturbations.

Figure 1: Architecture of the framework proposed for 3D volumes. The slices of each volume are first processed separately and then combined by applying an attention module over the slices. For each volume we also consider as input the corresponding saliency map. From the saliency branch, we obtain s

Experimental results

Research questions

• RQ1 Can saliency maps derived from adversarial counterfactuals effectively isolate small, diagnostically relevant regions in low-signal medical images without manual annotations?
• RQ2 Does multi-scale attention modulation using saliency-based masks improve classification accuracy in brain and lung CT scan tasks compared to baseline CNNs?
• RQ3 How does ACAT’s attention mechanism compare to random dropout or single-scale attention in terms of performance and robustness?
• RQ4 To what extent do attention masks generated via counterfactuals improve localization accuracy for lesion regions in multi-class localization tasks?
• RQ5 Is the performance gain of ACAT due to meaningful attention or merely a regularization effect from random feature dropping?

Key findings

• ACAT improves classification accuracy on IST-3 brain CT scans from 71.39% to 72.55%, a relative improvement of 1.16 percentage points.
• On MosMed lung CT scans, ACAT increases accuracy from 67.71% to 70.84%, a relative gain of 3.13 percentage points.
• ACAT achieves a lesion localization score of 65.05% on brain CT scans, outperforming the best competing method at 61.29%.
• Removing the attention fusion layer reduces accuracy to 69.79%, and removing all attention modules further degrades performance to 68.23%, demonstrating the importance of multi-scale attention.
• ACAT significantly reduces pre-activation variance across all network layers, especially under input noise, indicating improved robustness and smoother optimization.
• Random dropout experiments show that ACAT’s performance gain cannot be replicated by random feature dropping, confirming that the attention masks are informative and not merely regularizing.