[Paper Review] Cooperation Learning Enhanced Colonic Polyp Segmentation Based on Transformer-CNN Fusion

Yuanyuan Wang, Zhaohong Deng+4 more|arXiv (Cornell University)|Jan 17, 2023

Colorectal Cancer Screening and DetectionMedicine3 citations

TL;DR

This paper proposes Fu-TransHNet, a novel Transformer-CNN hybrid network for colonic polyp segmentation that enhances feature learning through multi-view cooperative learning and a dual-branch fusion module. It achieves state-of-the-art performance, with a 12.4% mDice improvement over HardNet-MSEG and 6.2% over TransFuse-s on the ETIS-LaribPolypDB dataset, particularly excelling in small polyp segmentation.

ABSTRACT

Traditional segmentation methods for colonic polyps are mainly designed based on low-level features. They could not accurately extract the location of small colonic polyps. Although the existing deep learning methods can improve the segmentation accuracy, their effects are still unsatisfied. To meet the above challenges, we propose a hybrid network called Fusion-Transformer-HardNetMSEG (i.e., Fu-TransHNet) in this study. Fu-TransHNet uses deep learning of different mechanisms to fuse each other and is enhanced with multi-view collaborative learning techniques. Firstly, the Fu-TransHNet utilizes the Transformer branch and the CNN branch to realize the global feature learning and local feature learning, respectively. Secondly, a fusion module is designed to integrate the features from two branches. The fusion module consists of two parts: 1) the Global-Local Feature Fusion (GLFF) part and 2) the Dense Fusion of Multi-scale features (DFM) part. The former is built to compensate the feature information mission from two branches at the same scale; the latter is constructed to enhance the feature representation. Thirdly, the above two branches and fusion modules utilize multi-view cooperative learning techniques to obtain their respective weights that denote their importance and then make a final decision comprehensively. Experimental results showed that the Fu-TransHNet network was superior to the existing methods on five widely used benchmark datasets. In particular, on the ETIS-LaribPolypDB dataset containing many small-target colonic polyps, the mDice obtained by Fu-TransHNet were 12.4% and 6.2% higher than the state-of-the-art methods HardNet-MSEG and TransFuse-s, respectively.

Motivation & Objective

• To address the challenge of inaccurate segmentation of small colonic polyps using traditional methods based on low-level features.
• To improve segmentation accuracy by fusing global context from Transformers with local detail from CNNs.
• To enhance feature representation through a multi-scale fusion module and multi-view cooperative learning.
• To achieve superior performance on benchmark datasets, especially for small polyp detection.

Proposed method

• The network employs a dual-branch architecture: a Transformer branch for global feature learning and a CNN branch for local feature learning.
• A Global-Local Feature Fusion (GLFF) module integrates features from both branches at the same scale to compensate for missing information.
• A Dense Fusion of Multi-scale features (DFM) module enhances feature representation by aggregating multi-scale features across branches.
• Multi-view cooperative learning is applied to dynamically assign importance weights to each branch and fusion module, enabling a comprehensive final decision.
• The fusion module combines attention-based global context and convolutional local patterns through weighted integration.
• The entire network is trained end-to-end with a segmentation loss function optimized via backpropagation.

Experimental results

Research questions

• RQ1 Can a hybrid Transformer-CNN architecture improve segmentation accuracy for small colonic polyps compared to existing methods?
• RQ2 How effectively can multi-view cooperative learning enhance feature fusion in polyp segmentation?
• RQ3 Does the proposed fusion module outperform standard concatenation or element-wise operations in preserving both global and local features?
• RQ4 To what extent does the model generalize across diverse benchmark datasets with varying polyp sizes and appearances?

Key findings

• On the ETIS-LaribPolypDB dataset, Fu-TransHNet achieved a 12.4% higher mDice score than the state-of-the-art method HardNet-MSEG.
• The model improved mDice by 6.2% over TransFuse-s on the same challenging dataset with numerous small polyps.
• Fu-TransHNet outperformed existing methods across five widely used benchmark datasets, demonstrating consistent generalization.
• The ablation study confirmed that both the GLFF and DFM modules significantly contributed to performance gains.
• The multi-view cooperative learning mechanism effectively balanced the contributions of the Transformer and CNN branches, improving robustness.