[Paper Review] ConvFormer: Combining CNN and Transformer for Medical Image Segmentation

Pengfei Gu, Yejia Zhang+2 more|arXiv (Cornell University)|Nov 15, 2022

Radiomics and Machine Learning in Medical Imaging4 citations

TL;DR

This paper proposes ConvFormer, a hierarchical CNN-Transformer hybrid architecture for medical image segmentation that integrates local inductive bias from convolutional networks with global long-range modeling from transformers. By introducing an Enhanced Deformable Transformer with a convolutional feed-forward module, a residual-shaped hybrid stem, and an additional multi-scale encoder, ConvFormer achieves state-of-the-art performance across 2D and 3D medical imaging datasets, outperforming existing CNN-only and Transformer-only models while maintaining efficient training from scratch.

ABSTRACT

Convolutional neural network (CNN) based methods have achieved great successes in medical image segmentation, but their capability to learn global representations is still limited due to using small effective receptive fields of convolution operations. Transformer based methods are capable of modelling long-range dependencies of information for capturing global representations, yet their ability to model local context is lacking. Integrating CNN and Transformer to learn both local and global representations while exploring multi-scale features is instrumental in further improving medical image segmentation. In this paper, we propose a hierarchical CNN and Transformer hybrid architecture, called ConvFormer, for medical image segmentation. ConvFormer is based on several simple yet effective designs. (1) A feed forward module of Deformable Transformer (DeTrans) is re-designed to introduce local information, called Enhanced DeTrans. (2) A residual-shaped hybrid stem based on a combination of convolutions and Enhanced DeTrans is developed to capture both local and global representations to enhance representation ability. (3) Our encoder utilizes the residual-shaped hybrid stem in a hierarchical manner to generate feature maps in different scales, and an additional Enhanced DeTrans encoder with residual connections is built to exploit multi-scale features with feature maps of different scales as input. Experiments on several datasets show that our ConvFormer, trained from scratch, outperforms various CNN- or Transformer-based architectures, achieving state-of-the-art performance.

Motivation & Objective

• To address the limitation of CNNs in capturing long-range dependencies due to small effective receptive fields.
• To overcome the lack of local inductive bias in pure Transformer models, which hinders performance in medical image segmentation.
• To improve multi-scale feature representation by integrating feature maps from different stages in a hierarchical encoder architecture.
• To enhance positional encoding flexibility for better adaptation to medical image data.
• To design a unified, end-to-end trainable architecture that outperforms existing CNN- and Transformer-based methods in segmentation accuracy.

Proposed method

• Redesign the feed-forward module of Deformable Transformer (DeTrans) with a convolutional layer to enhance local feature learning, creating the Enhanced DeTrans module.
• Develop a residual-shaped hybrid stem that combines depthwise separable convolutions and Enhanced DeTrans to jointly capture local and global representations at multiple scales.
• Construct a hierarchical encoder using the hybrid stem to generate multi-scale feature maps, which are then fed into an additional Enhanced DeTrans encoder with residual connections to exploit cross-scale contextual information.
• Introduce an Enhanced Positional Encoding (EPE) by adding a convolutional layer to the fixed sinusoidal positional encoding to improve adaptability and flexibility in feature representation.
• Utilize a U-Net-style decoder with skip connections to refine segmentation masks using features from the encoder.
• Train the entire network end-to-end from scratch on public and in-house datasets without pretraining.

Fig. 1 : (a) An overview of our Enhanced DeTrans with the proposed enhanced positional encoding (EPE) and Conv-based feed forward module (FFM) (shown in gray color). (b) Our ConvFormer: The two big boxes on the left are for the encoder and decoder, respectively (best viewed in color).

Experimental results

Research questions

• RQ1 Can a hybrid architecture combining CNN and Transformer improve medical image segmentation by balancing local and global feature learning?
• RQ2 Does introducing a convolutional feed-forward module into DeTrans enhance its ability to model local context in medical images?
• RQ3 To what extent does a hierarchical encoder with multi-scale feature input improve segmentation performance compared to standard single-scale encoders?
• RQ4 How effective is the proposed Enhanced Positional Encoding in improving model generalization and feature representation?
• RQ5 Can a unified CNN-Transformer architecture trained from scratch achieve state-of-the-art performance on diverse medical imaging tasks?

Key findings

• On the 2017 ISIC skin lesion dataset, ConvFormer achieved a Dice score of 0.955 and IoU of 0.914, outperforming all compared methods including UNETR and CoTr.
• On the 2017 MM-WHS CT dataset, ConvFormer achieved a Dice score of 0.980 and IoU of 0.957, demonstrating superior performance in 3D abdominal organ segmentation.
• On the in-house lymph node dataset, ConvFormer achieved a Dice score of 0.844 and IoU of 0.740, with a 0.031 improvement in Dice over the second-best method.
• Ablation studies confirmed that removing the Enhanced DeTrans module reduced F1 score by 0.4%, and removing the additional multi-scale encoder reduced it by 0.5%, validating their contributions.
• The removal of the hybrid stem (i.e., using only CNN) reduced F1 by 9.6%, demonstrating the critical role of joint local-global learning.
• The enhanced positional encoding (EPE) improved performance by 0.4% in F1 score compared to fixed sinusoidal encoding, confirming its effectiveness in feature representation.

Fig. 2 : Some visual qualitative results on the lymph node dataset (a) and the 2017 MM-WHS CT dataset (b), demonstrating the capability of our ConvFormer.