[Paper Review] Boosting Few-shot Semantic Segmentation with Transformers

Guolei Sun, Yun Liu+2 more|arXiv (Cornell University)|Aug 4, 2021

Advanced Neural Network Applications57 references4 citations

TL;DR

This paper proposes TRFS, a novel few-shot semantic segmentation framework that integrates global context modeling via Transformers and local feature refinement via convolutions. By combining a Global Enhancement Module (GEM) based on transformer blocks with a Local Enhancement Module (LEM) using convolutions, TRFS achieves state-of-the-art performance on PASCAL-5i and COCO benchmarks, demonstrating that global and local information are complementary for few-shot segmentation.

ABSTRACT

Due to the fact that fully supervised semantic segmentation methods require sufficient fully-labeled data to work well and can not generalize to unseen classes, few-shot segmentation has attracted lots of research attention. Previous arts extract features from support and query images, which are processed jointly before making predictions on query images. The whole process is based on convolutional neural networks (CNN), leading to the problem that only local information is used. In this paper, we propose a TRansformer-based Few-shot Semantic segmentation method (TRFS). Specifically, our model consists of two modules: Global Enhancement Module (GEM) and Local Enhancement Module (LEM). GEM adopts transformer blocks to exploit global information, while LEM utilizes conventional convolutions to exploit local information, across query and support features. Both GEM and LEM are complementary, helping to learn better feature representations for segmenting query images. Extensive experiments on PASCAL-5i and COCO datasets show that our approach achieves new state-of-the-art performance, demonstrating its effectiveness.

Motivation & Objective

• Address the limitation of existing few-shot semantic segmentation methods that rely solely on local receptive fields, missing long-range contextual relationships.
• Investigate the effectiveness of global context modeling via Transformers in few-shot segmentation, where only a few annotated support images are available.
• Demonstrate that combining global (transformer-based) and local (convolutional) feature refinement leads to superior performance compared to using either alone.
• Achieve state-of-the-art performance on standard few-shot segmentation benchmarks, including PASCAL-5i and COCO.

Proposed method

• Propose a two-branch architecture: Global Enhancement Module (GEM) that uses multi-head self-attention in transformer blocks to model long-range dependencies across query and support features.
• Introduce Local Enhancement Module (LEM) that applies standard convolutional layers to refine features at the pixel level, preserving local spatial details.
• Concatenate query features, support prototype features, and a prior mask into a single feature map of shape $ H \times W \times (2C+1) $, which is processed by both GEM and LEM in parallel.
• Apply multi-scale feature processing using adaptive average pooling at scales {60, 30, 15, 8} to enhance feature representation robustness across different resolutions.
• Train the model end-to-end using cross-entropy loss on the query segmentation predictions, with supervision from ground-truth masks.
• Use ResNet-50 as the backbone network for feature extraction, followed by feature concatenation and refinement through GEM and LEM.

Experimental results

Research questions

• RQ1 Can global context modeling via Transformers improve few-shot semantic segmentation performance when only a few support images are available?
• RQ2 Is the global information captured by Transformers complementary to local features extracted via convolutions in few-shot segmentation?
• RQ3 How does the number of transformer layers and multi-scale processing affect the model’s performance and generalization on novel classes?
• RQ4 Does combining global and local feature refinement lead to better segmentation accuracy than using either module in isolation?

Key findings

• The proposed TRFS model achieves a mean mIoU of 61.9 on the 1-shot setting of the PASCAL-5i dataset, outperforming previous state-of-the-art methods.
• The ablation study shows that using only GEM or only LEM achieves similar average mIoU (60.6 and 60.8, respectively), but their combination yields 61.9, proving their complementary nature.
• The model with 3 transformer blocks (L=3) achieves the best performance, indicating that deeper networks may overfit to base classes and degrade performance on novel classes.
• Multi-scale processing improves performance incrementally: from 58.6 (single scale) to 61.9 (four scales), with the full scale set {60, 30, 15, 8} used in all reported results.
• Qualitative results show that TRFS produces accurate segmentation masks on novel, unseen classes even with only one support image and its mask.
• On the COCO dataset, TRFS achieves state-of-the-art performance, confirming its generalization capability beyond the PASCAL-5i benchmark.