[Paper Review] Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers

Bo Dong, Wenhai Wang+4 more|arXiv (Cornell University)|Aug 16, 2021

Advanced Neural Network Applications120 references170 citations

TL;DR

Polyp-PVT introduces a pyramid vision transformer encoder with three add-on modules (CFM, CIM, SAM) to improve polyp segmentation, achieving state-of-the-art or competitive Dice scores across multiple benchmarks.

ABSTRACT

Most polyp segmentation methods use CNNs as their backbone, leading to two key issues when exchanging information between the encoder and decoder: 1) taking into account the differences in contribution between different-level features and 2) designing an effective mechanism for fusing these features. Unlike existing CNN-based methods, we adopt a transformer encoder, which learns more powerful and robust representations. In addition, considering the image acquisition influence and elusive properties of polyps, we introduce three standard modules, including a cascaded fusion module (CFM), a camouflage identification module (CIM), and a similarity aggregation module (SAM). Among these, the CFM is used to collect the semantic and location information of polyps from high-level features; the CIM is applied to capture polyp information disguised in low-level features, and the SAM extends the pixel features of the polyp area with high-level semantic position information to the entire polyp area, thereby effectively fusing cross-level features. The proposed model, named Polyp-PVT, effectively suppresses noises in the features and significantly improves their expressive capabilities. Extensive experiments on five widely adopted datasets show that the proposed model is more robust to various challenging situations (e.g., appearance changes, small objects, rotation) than existing representative methods. The proposed model is available at https://github.com/DengPingFan/Polyp-PVT.

Motivation & Objective

• Address limitations of CNN backbones in cross-level feature fusion for polyp segmentation.
• Introduce a transformer-based encoder (PVT) to learn robust multi-scale representations.
• Propose three modules (CFM, CIM, SAM) to fuse high- and low-level features and suppress noise.
• Evaluate Polyp-PVT on five challenging datasets and compare with state-of-the-art methods.

Proposed method

• Adopt a Pyramid Vision Transformer (PVTv2) as the encoder to extract multi-scale features X1–X4 from input images.
• Use Cascaded Fusion Module (CFM) to progressively fuse high-level features and produce T1.
• Apply Camouflage Identification Module (CIM) to enhance low-level features X1 into T2 via channel and spatial attention.
• Introduce Similarity Aggregation Module (SAM) combining non-local and graph convolutional operations to fuse T1 and T2 into final feature Z.
• Predict segmentation via a 1x1 conv head; train with a main loss (IoU + BCE) and an auxiliary loss for intermediate outputs.

Experimental results

Research questions

• RQ1 How does Polyp-PVT perform on polyp segmentation compared to CNN-based backbones across standard benchmarks?
• RQ2 What is the contribution of CFM, CIM, and SAM to overall performance and robustness to challenging conditions (noise, camouflage, cross-domain data)?
• RQ3 How does the transformer-based encoder cope with appearance changes, small polyps, and rotations in endoscopic imagery?

Key findings

• Polyp-PVT achieves strong cross-dataset performance, e.g., on Kvasir-SEG (mDic 0.917) and ClinicDB (mDic 0.937).
• On ColonDB, Polyp-PVT achieves mDic 0.808, outperforming SANet by a margin (as reported).
• On ETIS, Polyp-PVT achieves mDic 0.787, surpassing SANet by a notable margin.
• On Endoscene, Polyp-PVT reaches mDic 0.900 and mIoU 0.833, indicating robust performance in challenging conditions.
• Overall, Polyp-PVT demonstrates robustness to appearance changes, small objects, and rotations, outperforming several representative baselines such as SANet and PraNet.