[Paper Review] Dual Cross-Attention for Medical Image Segmentation

Gorkem Can Ates, Prasoon P. Mohan+1 more|arXiv (Cornell University)|Mar 30, 2023

Radiomics and Machine Learning in Medical Imaging74 references4 citations

TL;DR

This paper proposes Dual Cross-Attention (DCA), a lightweight attention module that enhances skip-connections in U-Net-based medical image segmentation models by sequentially modeling channel-wise and spatial dependencies across multi-scale encoder features using cross-attention. DCA achieves consistent Dice score improvements of up to 2.05% across five benchmark datasets by narrowing the semantic gap between encoder and decoder features without increasing model complexity significantly.

ABSTRACT

We propose Dual Cross-Attention (DCA), a simple yet effective attention module that is able to enhance skip-connections in U-Net-based architectures for medical image segmentation. DCA addresses the semantic gap between encoder and decoder features by sequentially capturing channel and spatial dependencies across multi-scale encoder features. First, the Channel Cross-Attention (CCA) extracts global channel-wise dependencies by utilizing cross-attention across channel tokens of multi-scale encoder features. Then, the Spatial Cross-Attention (SCA) module performs cross-attention to capture spatial dependencies across spatial tokens. Finally, these fine-grained encoder features are up-sampled and connected to their corresponding decoder parts to form the skip-connection scheme. Our proposed DCA module can be integrated into any encoder-decoder architecture with skip-connections such as U-Net and its variants. We test our DCA module by integrating it into six U-Net-based architectures such as U-Net, V-Net, R2Unet, ResUnet++, DoubleUnet and MultiResUnet. Our DCA module shows Dice Score improvements up to 2.05% on GlaS, 2.74% on MoNuSeg, 1.37% on CVC-ClinicDB, 1.12% on Kvasir-Seg and 1.44% on Synapse datasets. Our codes are available at: https://github.com/gorkemcanates/Dual-Cross-Attention

Motivation & Objective

• To address the semantic gap between encoder and decoder features in U-Net-based architectures for medical image segmentation.
• To improve skip-connection performance by modeling long-range dependencies across multi-scale encoder features.
• To design a lightweight, plug-and-play attention module that enhances feature fusion without increasing computational overhead.
• To validate the effectiveness of DCA across diverse U-Net variants and benchmark datasets.

Proposed method

• DCA employs Channel Cross-Attention (CCA) to capture global channel-wise dependencies by applying cross-attention across channel tokens of multi-scale encoder features.
• Spatial Cross-Attention (SCA) is applied afterward to model spatial dependencies by cross-attending over spatial tokens in the feature maps.
• The CCA and SCA modules are applied sequentially in the CCA-SCA order, with feature maps processed through 2D average pooling for patch embedding and 1×1 depth-wise convolutions for projection.
• The enhanced encoder features are upsampled and connected to their corresponding decoder layers via skip-connections, replacing standard concatenation.
• The DCA block is designed to be easily integrated into any U-Net-based architecture with skip-connections.
• Ablation studies compare fusion strategies (sum, concat, sequential) and patch embedding methods (average pooling vs. convolution).

Experimental results

Research questions

• RQ1 Does a dual-attention mechanism that sequentially models channel and spatial dependencies improve feature representation in U-Net-based segmentation?
• RQ2 Which order of channel and spatial cross-attention (CCA-SCA vs. SCA-CCA) yields optimal performance?
• RQ3 Which fusion strategy (sum, concat, sequential) best combines channel and spatial attention outputs?
• RQ4 How does average pooling compare to convolutional patch embedding in terms of performance and parameter efficiency?

Key findings

• The CCA-SCA sequential layout achieves the best performance, improving Dice scores by 0.79% on GlaS and 0.16% on MoNuSeg compared to individual modules.
• Sequential fusion (CCA-then-SCA) outperforms both summation and concatenation strategies, achieving a 2.05% Dice score improvement on the GlaS dataset.
• Using 2D average pooling for patch embedding yields better performance than convolutional patch embedding, with a 0.14% higher Dice score on GlaS and reduced parameter count.
• DCA improves segmentation performance across six U-Net variants, with gains of 2.74% on MoNuSeg, 1.44% on Synapse, and 1.37% on CVC-ClinicDB.
• The DCA module maintains low computational cost, with only a minor increase in parameters (e.g., 8.75M vs. 8.64M in U-Net) while delivering consistent gains.
• The method generalizes well, showing significant improvements on diverse datasets including GlaS, MoNuSeg, CVC-ClinicDB, Kvasir-Seg, and Synapse.