[Paper Review] Fully Transformer Networks for Semantic Image Segmentation

Sitong Wu, Tianyi Wu+3 more|arXiv (Cornell University)|Jun 8, 2021

Advanced Neural Network Applications60 references32 citations

TL;DR

This paper presents Fully Transformer Networks (FTN) that use a Pyramid Group Transformer encoder (PGT) and a Feature Pyramid Transformer decoder (FPT) to achieve state-of-the-art semantic segmentation without CNNs, across PASCAL Context, ADE20K, COCO-Stuff, and CelebAMask-HQ.

ABSTRACT

Transformers have shown impressive performance in various natural language processing and computer vision tasks, due to the capability of modeling long-range dependencies. Recent progress has demonstrated that combining such Transformers with CNN-based semantic image segmentation models is very promising. However, it is not well studied yet on how well a pure Transformer based approach can achieve for image segmentation. In this work, we explore a novel framework for semantic image segmentation, which is encoder-decoder based Fully Transformer Networks (FTN). Specifically, we first propose a Pyramid Group Transformer (PGT) as the encoder for progressively learning hierarchical features, meanwhile reducing the computation complexity of the standard Visual Transformer (ViT). Then, we propose a Feature Pyramid Transformer (FPT) to fuse semantic-level and spatial-level information from multiple levels of the PGT encoder for semantic image segmentation. Surprisingly, this simple baseline can achieve better results on multiple challenging semantic segmentation and face parsing benchmarks, including PASCAL Context, ADE20K, COCOStuff, and CelebAMask-HQ. The source code will be released on https://github.com/BR-IDL/PaddleViT.

Motivation & Objective

• Motivate exploration of pure Transformer architectures for pixel-level semantic segmentation without CNN components.
• Introduce a hierarchical Transformer encoder (PGT) to learn multi-scale representations with controllable receptive fields.
• Propose a Transformer-based decoder (FPT) to fuse semantic and spatial information across levels.
• Demonstrate state-of-the-art performance on standard segmentation benchmarks.

Proposed method

• Define Pyramid Group Transformer (PGT) as a four-stage encoder with patch transforms and Pyramid Group Multi-Head Self-Attention (PG-MSA) to learn hierarchical features.
• Control receptive fields by organizing attention within non-overlapping groups that progressively increase across stages.
• Introduce Feature Pyramid Transformer (FPT) as a top-down, multi-level fusion decoder with lateral connections and Transformer blocks to assemble high-resolution predictions.
• Train FTN in a fully Transformer-based encoder-decoder framework and evaluate on PASCAL Context, ADE20K, COCO-Stuff, and CelebAMask-HQ.
• Pretrain PGT on ImageNet-1K and fine-tune on segmentation benchmarks; apply standard data augmentation and training schedules.

Experimental results

Research questions

• RQ1 Can a fully Transformer-based encoder-decoder framework match or exceed CNN-backed segmentation models on standard benchmarks?
• RQ2 Does a pyramid/grouped self-attention encoder with a Transformer-based decoder effectively capture multi-scale context for pixel-level prediction?
• RQ3 What are the impacts of encoder/decoder choices and multi-scale fusion strategies on segmentation accuracy?
• RQ4 How does FTN compare to state-of-the-art Transformer and CNN-based segmentation methods in terms of accuracy and efficiency?

Key findings

• FTN with PGT and FPT achieves state-of-the-art or competitive mIoU on major benchmarks: 56.05% (PASCAL Context), 51.36% (ADE20K), and 45.89% (COCO-Stuff) for FTN-L.
• FTN-T, FTN-S, FTN-B, FTN-L outperform comparable backbones (PVT, Swin, ViT) under similar computation, with FTN-L surpassing ViT-L/16 in some settings.
• The Pyramid Group Transformer (PGT) learns hierarchical features and reduces computation/memory versus global ViT, enabling effective dense prediction.
• The Feature Pyramid Transformer (FPT) effectively fuses multi-level semantic and spatial information, yielding consistent gains over other decoders.
• FTN variants demonstrate strong performance on CelebAMask-HQ for face parsing, with FTN-L achieving 87.4 mean F1 score and outperforming several baselines.
• Pretraining on ImageNet-1K is sufficient for competitive results, with improvements observed when using larger backbones and multi-scale inference.