[Paper Review] TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation

Yundong Zhang, Huiye Liu+1 more|arXiv (Cornell University)|Feb 16, 2021

Advanced Neural Network ApplicationsComputer Science38 references101 citations

TL;DR

TransFuse introduces a parallel CNN and Transformer architecture with a BiFusion fusion module to jointly capture low-level details and global context for medical image segmentation, achieving state-of-the-art results with fewer parameters and faster inference.

ABSTRACT

Medical image segmentation - the prerequisite of numerous clinical needs - has been significantly prospered by recent advances in convolutional neural networks (CNNs). However, it exhibits general limitations on modeling explicit long-range relation, and existing cures, resorting to building deep encoders along with aggressive downsampling operations, leads to redundant deepened networks and loss of localized details. Hence, the segmentation task awaits a better solution to improve the efficiency of modeling global contexts while maintaining a strong grasp of low-level details. In this paper, we propose a novel parallel-in-branch architecture, TransFuse, to address this challenge. TransFuse combines Transformers and CNNs in a parallel style, where both global dependency and low-level spatial details can be efficiently captured in a much shallower manner. Besides, a novel fusion technique - BiFusion module is created to efficiently fuse the multi-level features from both branches. Extensive experiments demonstrate that TransFuse achieves the newest state-of-the-art results on both 2D and 3D medical image sets including polyp, skin lesion, hip, and prostate segmentation, with significant parameter decrease and inference speed improvement.

Motivation & Objective

• Address the limitations of CNNs in modeling global context and preserving local details in medical image segmentation.
• Propose a parallel CNN-Transformer architecture (TransFuse) with a BiFusion module for efficient feature fusion.
• Show that parallel branches with BiFusion yield strong segmentation performance with fewer parameters and faster inference.
• Demonstrate state-of-the-art results on diverse 2D/3D medical datasets (polyp, skin lesion, hip, prostate).

Proposed method

• Two parallel branches process information differently: a CNN branch that emphasizes local spatial details and a Transformer branch that models global context.
• A BiFusion module fuses multi-level features from both branches using channel and spatial attention plus a Hadamard product.
• Late fusion with attention-based skip connections aggregates fused features to generate segmentation maps.
• Loss combines weighted IoU and weighted binary cross-entropy with deep supervision across branches.
• Variants (TransFuse-S, TransFuse-L, TransFuse-L*) explore different backbone choices (CNN + DeiT/ViT backbones).

Experimental results

Research questions

• RQ1 Can parallel CNN and Transformer branches capture complementary information for medical image segmentation without very deep networks?
• RQ2 How effective is the BiFusion module at fusing multi-level CNN and Transformer features?
• RQ3 Do the proposed TransFuse variants improve accuracy and efficiency across diverse 2D/3D medical segmentation tasks?
• RQ4 What is the performance of TransFuse on polyp, skin lesion, hip, and prostate segmentation datasets?

Key findings

• TransFuse achieves state-of-the-art results on multiple 2D and 3D medical segmentation tasks (polyp, skin lesion, hip, prostate).
• The single-branch CNN downsampling is reduced by leveraging a parallel Transformer branch, yielding a shallower overall model with improved global context.
• TransFuse-S attains competitive or superior results with approximately 26.3M parameters and high inference speed (e.g., 98.7 FPS on a RTX 2080 Ti).
• TransFuse-L* further improves performance over TransUnet on challenging polyp datasets, while maintaining favorable efficiency (e.g., about 45.3 FPS).
• ISIC 2017 skin lesion results show TransFuse-S achieving strong metrics (e.g., Jaccard/Dice around 0.795/0.872 with pretrained backbones).
• Ablation studies indicate the BiFusion module and the parallel-in-branch design contribute to performance gains over sequential fusion and simple concatenation strategies.