[Paper Review] ColonFormer: An Efficient Transformer based Method for Colon Polyp Segmentation

Nguyen Thanh Duc, Thi-Oanh Nguyen+3 more|arXiv (Cornell University)|May 17, 2022

Colorectal Cancer Screening and Detection4 citations

TL;DR

ColonFormer is a novel, efficient Transformer-based encoder-decoder network for colon polyp segmentation that combines a lightweight hierarchical Transformer encoder for multi-scale global feature learning with a CNN-based hierarchical decoder and a refinement module using residual-connected axial attention to enhance boundary precision. It achieves state-of-the-art performance across five benchmark datasets, outperforming existing methods in both mDice and mIOU metrics.

ABSTRACT

Identifying polyps is challenging for automatic analysis of endoscopic images in computer-aided clinical support systems. Models based on convolutional networks (CNN), transformers, and their combinations have been proposed to segment polyps with promising results. However, those approaches have limitations either in modeling the local appearance of the polyps only or lack of multi-level features for spatial dependency in the decoding process. This paper proposes a novel network, namely ColonFormer, to address these limitations. ColonFormer is an encoder-decoder architecture capable of modeling long-range semantic information at both encoder and decoder branches. The encoder is a lightweight architecture based on transformers for modeling global semantic relations at multi scales. The decoder is a hierarchical network structure designed for learning multi-level features to enrich feature representation. Besides, a refinement module is added with a new skip connection technique to refine the boundary of polyp objects in the global map for accurate segmentation. Extensive experiments have been conducted on five popular benchmark datasets for polyp segmentation, including Kvasir, CVC-Clinic DB, CVC-ColonDB, CVC-T, and ETIS-Larib. Experimental results show that our ColonFormer outperforms other state-of-the-art methods on all benchmark datasets.

Motivation & Objective

• To address the limitations of existing polyp segmentation models that either focus only on local features or lack multi-level spatial dependency modeling.
• To improve segmentation accuracy, especially at polyp boundaries, which are often ambiguous due to poor contrast and irregular shapes.
• To develop an efficient architecture that balances high performance with reduced computational cost for clinical deployment.
• To integrate global context modeling via self-attention with hierarchical feature learning for robust polyp detection across diverse imaging conditions.

Proposed method

• The encoder uses a lightweight, hierarchical Transformer architecture to capture multi-scale global semantic relationships across the input image.
• The decoder employs a UPer-like pyramid structure with hierarchical feature fusion to enrich multi-level representations for improved spatial understanding.
• A novel refinement module is introduced, incorporating axial attention with residual connections to iteratively refine the segmentation map and enhance boundary accuracy.
• The model uses a hybrid encoder-decoder design, combining the global modeling strength of Transformers with the inductive bias of CNNs for feature learning.
• The refinement module applies a skip connection mechanism that propagates coarse predictions back through the network to correct boundary errors.
• The architecture is optimized for efficiency, with ablation studies showing that the UPer decoder reduces GFLOPs by over 30% compared to MLP-based decoders.

Experimental results

Research questions

• RQ1 Can a lightweight Transformer-based encoder effectively model long-range dependencies and multi-scale features for colon polyp segmentation?
• RQ2 How does the integration of a hierarchical CNN decoder improve feature representation compared to standard decoders?
• RQ3 To what extent does the proposed refinement module with axial attention and residual connections improve boundary segmentation accuracy?
• RQ4 How does the overall ColonFormer architecture compare in performance and efficiency to state-of-the-art models on diverse polyp segmentation benchmarks?
• RQ5 What is the optimal trade-off between model size, computational cost, and segmentation accuracy across different backbone variants?

Key findings

• ColonFormer-S achieved the highest average mDice of 0.854 across all five benchmark datasets, outperforming all compared state-of-the-art methods.
• ColonFormer-L achieved an mDice of 0.801 and mIOU of 0.722 on the ETIS-Larib dataset, significantly outperforming previous models.
• The UPer decoder reduced GFLOPs by 33% compared to the MLP decoder while maintaining comparable performance, validating its efficiency.
• The refinement module improved mDice by up to 0.008 on the Kvasir and ETIS-Larib datasets, demonstrating its effectiveness in boundary refinement.
• ColonFormer-L achieved an mDice of 0.932 and mIOU of 0.884 on CVC-ClinicDB, exceeding the performance of SegFormer-B3-Uper-ARA by 0.010 and 0.009 respectively.
• The ablation study confirmed that the combination of UPer decoder and the refined axial attention module yielded the best overall performance, with minimal computational overhead.