[论文解读] AdaFuse: Adaptive Medical Image Fusion Based on Spatial-Frequential Cross Attention

Xian‐Ming Gu, Lihui Wang+4 more|arXiv (Cornell University)|Oct 9, 2023

Advanced Image Fusion TechniquesEngineering被引用 3

一句话总结

AdaFuse 提出了一种新颖的自适应医学图像融合框架，通过傅里叶变换实现空间-频率域交叉注意力，动态融合多模态医学图像。通过在空间域和频率域整合交叉注意力机制，并采用结合内容损失与结构损失的混合损失函数，该方法在保持细节和减少伪影方面表现出色，优于基准数据集上的最先进方法。

ABSTRACT

Multi-modal medical image fusion is essential for the precise clinical diagnosis and surgical navigation since it can merge the complementary information in multi-modalities into a single image. The quality of the fused image depends on the extracted single modality features as well as the fusion rules for multi-modal information. Existing deep learning-based fusion methods can fully exploit the semantic features of each modality, they cannot distinguish the effective low and high frequency information of each modality and fuse them adaptively. To address this issue, we propose AdaFuse, in which multimodal image information is fused adaptively through frequency-guided attention mechanism based on Fourier transform. Specifically, we propose the cross-attention fusion (CAF) block, which adaptively fuses features of two modalities in the spatial and frequency domains by exchanging key and query values, and then calculates the cross-attention scores between the spatial and frequency features to further guide the spatial-frequential information fusion. The CAF block enhances the high-frequency features of the different modalities so that the details in the fused images can be retained. Moreover, we design a novel loss function composed of structure loss and content loss to preserve both low and high frequency information. Extensive comparison experiments on several datasets demonstrate that the proposed method outperforms state-of-the-art methods in terms of both visual quality and quantitative metrics. The ablation experiments also validate the effectiveness of the proposed loss and fusion strategy.

研究动机与目标

• 为解决现有基于深度学习的融合方法在自适应区分并融合多模态医学图像中低频与高频分量方面的局限性。
• 克服依赖人工设计的融合规则与特征提取技术所导致的鲁棒性与自适应性降低的问题。
• 通过频率引导的注意力机制，增强融合后医学图像中高频细节与结构完整性的保留。
• 开发一种可学习的端到端融合框架，避免复杂的人工设计，同时保持高性能。
• 验证一种结合内容损失与结构损失的新型多损失函数在提升融合质量方面的有效性。

提出的方法

• 提出一种交叉注意力融合（CAF）模块，通过在空间域与频率域之间交换查询与键向量，实现两模态之间的自适应特征融合。
• 应用傅里叶变换提取频域表征，实现频率引导的注意力机制，以提升高频细节的保留能力。
• 在空间特征与频率特征之间引入交叉注意力，进一步通过域间信息流引导优化融合过程。
• 采用多尺度架构，在四个尺度上堆叠 Transformer 编码器，以捕捉多分辨率下的层次化特征。
• 设计一种混合损失函数，结合结构损失与内容损失，以在融合输出中同时保留低频与高频信息。
• 在 CAF 模块内引入自注意力机制，按图像块动态计算自适应融合权重，增强特征重要性建模能力。

Figure 1: Examples of medical images from different modalities. (a)CT images can provide bone density information; (b)MRI images can provide high-contrast and high-resolution soft tissue structures; (c)PET and (d)SPECT can reflect physiological activities such as metabolism, molecular motion, and bl

实验结果

研究问题

• RQ1 频率引导的交叉注意力机制是否能提升多模态医学图像中低频与高频分量的自适应融合？
• RQ2 空间域与频率域注意力的融合如何影响融合后医学图像的细节保留与图像质量？
• RQ3 所提出的混合损失函数在多大程度上增强了源图像中互补信息的保留？
• RQ4 CAF 模块在视觉质量与定量指标方面是否优于传统融合策略？
• RQ5 所提出方法是否能在不依赖人工融合规则或大规模标注数据的情况下实现更优性能？

主要发现

• AdaFuse 在四项定量指标（EN、MI、CC 和 FMI）上均达到最先进水平，证明其在保留源模态信息方面表现优异。
• 该方法获得较高的 PSNR 值，表明融合输出具有强结构完整性与最小的图像失真。
• 视觉结果表明边缘更清晰，模糊或对比度伪影更少，证实了有效细节保留。
• 消融实验表明，所提出的损失函数与 CAF 模块显著提升了融合质量；当移除模块时 EN 值下降，归因于伪影引入。
• 该方法在多个公开数据集上优于现有最先进方法，验证了其鲁棒性与泛化能力。
• 该框架在无需人工规则设计的情况下，成功融合了 CT、MRI、PET 与 SPECT 模态的互补信息，展现出强大的适应性。

Figure 2: Architecture of the proposed AdaFuse. Our method performs feature fusion at each of the four scales using a spatial-frequential fusion (SFF) module. In SFF, we designed the Fourier-guided fusion branch (FGFB) to extract the frequency domain features and proposed cross attention fusion (CAF