[Paper Review] ConvTransSeg: A Multi-resolution Convolution-Transformer Network for Medical Image Segmentation

Zhendi Gong, Andrew P. French+2 more|arXiv (Cornell University)|Oct 13, 2022

AI in cancer detection4 citations

TL;DR

This paper proposes ConvTransSeg, a hybrid medical image segmentation model that uses a multi-resolution convolutional neural network (CNN) encoder for local feature extraction and a multi-resolution Transformer decoder for long-range contextual modeling. The method achieves state-of-the-art performance on multiple public datasets with lower model complexity and without requiring pre-training, outperforming both CNN-only and Transformer-only baselines in Dice coefficient and average symmetric surface distance.

ABSTRACT

Convolutional neural networks (CNNs) achieved the state-of-the-art performance in medical image segmentation due to their ability to extract highly complex feature representations. However, it is argued in recent studies that traditional CNNs lack the intelligence to capture long-term dependencies of different image regions. Following the success of applying Transformer models on natural language processing tasks, the medical image segmentation field has also witnessed growing interest in utilizing Transformers, due to their ability to capture long-range contextual information. However, unlike CNNs, Transformers lack the ability to learn local feature representations. Thus, to fully utilize the advantages of both CNNs and Transformers, we propose a hybrid encoder-decoder segmentation model (ConvTransSeg). It consists of a multi-layer CNN as the encoder for feature learning and the corresponding multi-level Transformer as the decoder for segmentation prediction. The encoder and decoder are interconnected in a multi-resolution manner. We compared our method with many other state-of-the-art hybrid CNN and Transformer segmentation models on binary and multiple class image segmentation tasks using several public medical image datasets, including skin lesion, polyp, cell and brain tissue. The experimental results show that our method achieves overall the best performance in terms of Dice coefficient and average symmetric surface distance measures with low model complexity and memory consumption. In contrast to most Transformer-based methods that we compared, our method does not require the use of pre-trained models to achieve similar or better performance. The code is freely available for research purposes on Github: (the link will be added upon acceptance).

Motivation & Objective

• To address the limitation of CNNs in capturing long-range dependencies in medical image segmentation.
• To overcome the lack of local feature learning in pure Transformer models for small medical datasets.
• To design a hybrid architecture that combines the strengths of CNNs (local feature extraction) and Transformers (long-range context) in a multi-resolution framework.
• To develop a method that achieves high segmentation accuracy with low model complexity and memory consumption, without relying on pre-trained models.
• To enable flexible input size and efficient training for small medical image datasets.

Proposed method

• The model employs a multi-layer CNN as the encoder to extract hierarchical, multi-resolution local features from input images.
• The decoder is a multi-level Transformer that processes the encoded features using self-attention mechanisms to model long-range dependencies across image regions.
• Encoder and decoder features are connected via computationally efficient re-sampling and re-shaping operations at multiple scales.
• The architecture is designed to maintain feature resolution throughout the network, enabling effective fusion of local and global context.
• The model is trained end-to-end using standard segmentation loss functions, with no need for pre-training on large-scale datasets.
• Multi-resolution feature interaction is achieved through skip-like connections that preserve spatial information across scales.

Experimental results

Research questions

• RQ1 Can a hybrid CNN-Transformer architecture outperform pure CNN and pure Transformer models in medical image segmentation tasks?
• RQ2 Does using a CNN encoder with a Transformer decoder improve performance on small medical datasets without requiring pre-training?
• RQ3 How does multi-resolution feature interaction between encoder and decoder affect segmentation accuracy and model efficiency?
• RQ4 Can the proposed method achieve state-of-the-art results in terms of Dice coefficient and average symmetric surface distance while maintaining low parameter count and memory usage?
• RQ5 How robust is the method to challenging cases with low contrast between foreground and background, such as polyp or skin lesion segmentation?

Key findings

• ConvTransSeg achieved the highest mean Dice coefficient across all evaluated datasets, including ISIC2018, CVC-ClinicDB, Pannuke, and OASIS-1, outperforming all compared methods.
• The model achieved the lowest average symmetric surface distance (ASSD) on all datasets, indicating superior boundary localization accuracy.
• On the CVC-ClinicDB polyp dataset, ConvTransSeg significantly outperformed ResUNet, which struggled with low-contrast cases, and matched or exceeded the performance of other Transformer-based models.
• The method achieved state-of-the-art performance with fewer learnable parameters and faster convergence than competing models, demonstrating high efficiency.
• Qualitative results showed that ConvTransSeg produced more accurate predictions than ResUNet and other Transformer-based models in challenging cases with similar intensity patterns between target and background.
• The model demonstrated robustness in detecting small or irregularly shaped regions, such as polyps, even when other methods failed or produced over-segmentation.