[Paper Review] AdaFuse: Adaptive Medical Image Fusion Based on Spatial-Frequential Cross Attention

Xian‐Ming Gu, Lihui Wang+4 more|arXiv (Cornell University)|Oct 9, 2023

Advanced Image Fusion TechniquesEngineering3 citations

TL;DR

AdaFuse proposes a novel adaptive medical image fusion framework that leverages spatial-frequential cross attention via Fourier transform to dynamically fuse multi-modal medical images. By integrating cross-attention mechanisms in both spatial and frequency domains and employing a hybrid loss combining content and structure losses, the method achieves superior fusion quality with enhanced detail preservation and reduced artifacts, outperforming state-of-the-art methods on benchmark datasets.

ABSTRACT

Multi-modal medical image fusion is essential for the precise clinical diagnosis and surgical navigation since it can merge the complementary information in multi-modalities into a single image. The quality of the fused image depends on the extracted single modality features as well as the fusion rules for multi-modal information. Existing deep learning-based fusion methods can fully exploit the semantic features of each modality, they cannot distinguish the effective low and high frequency information of each modality and fuse them adaptively. To address this issue, we propose AdaFuse, in which multimodal image information is fused adaptively through frequency-guided attention mechanism based on Fourier transform. Specifically, we propose the cross-attention fusion (CAF) block, which adaptively fuses features of two modalities in the spatial and frequency domains by exchanging key and query values, and then calculates the cross-attention scores between the spatial and frequency features to further guide the spatial-frequential information fusion. The CAF block enhances the high-frequency features of the different modalities so that the details in the fused images can be retained. Moreover, we design a novel loss function composed of structure loss and content loss to preserve both low and high frequency information. Extensive comparison experiments on several datasets demonstrate that the proposed method outperforms state-of-the-art methods in terms of both visual quality and quantitative metrics. The ablation experiments also validate the effectiveness of the proposed loss and fusion strategy.

Motivation & Objective

• To address the limitation of existing deep learning-based fusion methods in adaptively distinguishing and fusing low- and high-frequency components from multi-modal medical images.
• To overcome the reliance on manually designed fusion rules and feature extraction techniques that reduce robustness and adaptiveness.
• To enhance the preservation of high-frequency details and structural integrity in fused medical images through frequency-guided attention mechanisms.
• To develop a learnable, end-to-end fusion framework that avoids complex manual design while maintaining high performance.
• To validate the effectiveness of a novel multi-loss function combining content and structure losses for improved fusion quality.

Proposed method

• Proposes a Cross-Attention Fusion (CAF) block that enables adaptive feature fusion between two modalities by exchanging query and key vectors across spatial and frequency domains.
• Applies Fourier transform to extract frequency-domain representations, enabling frequency-guided attention for improved high-frequency detail retention.
• Introduces cross-attention between spatial and frequency features to further refine fusion by guiding information flow between domains.
• Employs a multi-scale architecture with stacked Transformer encoders at four scales to capture hierarchical features across resolutions.
• Designs a hybrid loss function combining structure loss and content loss to preserve both low- and high-frequency information in the fused output.
• Uses a self-attention mechanism within the CAF block to dynamically compute adaptive fusion weights per patch, enhancing feature importance modeling.

Figure 1: Examples of medical images from different modalities. (a)CT images can provide bone density information; (b)MRI images can provide high-contrast and high-resolution soft tissue structures; (c)PET and (d)SPECT can reflect physiological activities such as metabolism, molecular motion, and bl

Experimental results

Research questions

• RQ1 Can frequency-guided cross-attention mechanisms improve the adaptive fusion of low- and high-frequency components in multi-modal medical images?
• RQ2 How does the integration of spatial and frequency domain attention affect detail preservation and image quality in fused medical images?
• RQ3 To what extent does the proposed hybrid loss function enhance the retention of complementary information from source images?
• RQ4 Does the CAF block outperform conventional fusion strategies in terms of both visual quality and quantitative metrics?
• RQ5 Can the proposed method achieve superior performance without relying on manual fusion rule design or large-scale labeled data?

Key findings

• AdaFuse achieves state-of-the-art performance across four quantitative metrics—EN, MI, CC, and FMI—demonstrating superior information retention from source modalities.
• The method achieves high PSNR values, indicating strong structural integrity and minimal image distortion in the fused output.
• Visual results show clearer edges and reduced blurring or contrast artifacts, confirming effective detail preservation.
• Ablation studies confirm that the proposed loss function and CAF block significantly improve fusion quality, with increased EN when modules are removed due to artifact introduction.
• The method outperforms existing SOTA methods on multiple public datasets, validating its robustness and generalization capability.
• The framework demonstrates strong adaptability in fusing complementary information from CT, MRI, PET, and SPECT modalities without manual rule design.

Figure 2: Architecture of the proposed AdaFuse. Our method performs feature fusion at each of the four scales using a spatial-frequential fusion (SFF) module. In SFF, we designed the Fourier-guided fusion branch (FGFB) to extract the frequency domain features and proposed cross attention fusion (CAF