[Paper Review] GFF: Gated Fully Fusion for Semantic Segmentation

Xiangtai Li, Houlong Zhao+3 more|arXiv (Cornell University)|Apr 3, 2019

Advanced Neural Network Applications56 references35 citations

TL;DR

Introduces Gated Fully Fusion (GFF) to selectively fuse multi-level features with gates, enhanced by a Dense Feature Pyramid, achieving state-of-the-art semantic segmentation on Cityscapes, Pascal Context, COCO-stuff, and ADE20K.

ABSTRACT

Semantic segmentation generates comprehensive understanding of scenes through densely predicting the category for each pixel. High-level features from Deep Convolutional Neural Networks already demonstrate their effectiveness in semantic segmentation tasks, however the coarse resolution of high-level features often leads to inferior results for small/thin objects where detailed information is important. It is natural to consider importing low level features to compensate for the lost detailed information in high-level features.Unfortunately, simply combining multi-level features suffers from the semantic gap among them. In this paper, we propose a new architecture, named Gated Fully Fusion (GFF), to selectively fuse features from multiple levels using gates in a fully connected way. Specifically, features at each level are enhanced by higher-level features with stronger semantics and lower-level features with more details, and gates are used to control the propagation of useful information which significantly reduces the noises during fusion. We achieve the state of the art results on four challenging scene parsing datasets including Cityscapes, Pascal Context, COCO-stuff and ADE20K.

Motivation & Objective

• Motivate improved semantic segmentation by leveraging both high-level semantics and high-resolution details.
• Develop a fusion mechanism that selectively propagates information across multiple feature levels.
• Incorporate context modeling to enhance semantic representation of multi-level features.
• Demonstrate state-of-the-art performance on multiple standard scene parsing benchmarks.

Proposed method

• Propose Gated Fully Fusion (GFF) to fuse multi-level features via per-pixel gates that control information propagation.
• Formulate gate maps G_l as sigmoid(w_l * X_l) to regulate sender and receiver information during fusion.
• Define the fusion at each level l as a gated addition: X̃_l = (1+G_l)·X_l + (1−G_l)·∑_{i≠l} G_i·X_i.
• Introduce a Dense Feature Pyramid (DFP) that densely connects PSPNet-style context to all feature maps to enrich contextual cues.
• Train end-to-end with a main segmentation loss plus an auxiliary loss on an intermediate ResNet stage to stabilize optimization.
• Optionally extend the backbone fusion with a top-down gated pathway and evaluate computational cost alongside accuracy.

Experimental results

Research questions

• RQ1 Does gated, fully connected fusion of multi-level features improve segmentation accuracy over traditional local or top-down fusion methods?
• RQ2 Can per-pixel gates effectively suppress noisy information during fusion to preserve detail for small/thin objects?
• RQ3 How does adding a Dense Feature Pyramid (DFP) affect contextual modeling and final segmentation performance?
• RQ4 Are the improvements from GFF and DFP consistent across diverse datasets and backbones?
• RQ5 What is the trade-off between accuracy gains and computational cost when integrating GFF and DFP into existing architectures?

Key findings

• GFF consistently improves over PSPNet baselines and other fusion variants on Cityscapes validation, achieving 80.4% mIoU for PSPNet + GFF.
• Adding DFP further boosts performance to 81.2% with GFF + DFP, and 81.8% with multi-scale inference (MS).
• With a ResNet101 backbone, the method reaches 82.3% mIoU on Cityscapes test set, and 83.3% mIoU with WiderResNet when trained on fine data only.
• GFF demonstrates superior handling of small/thin objects and object boundaries compared with traditional fusion methods like concatenation, addition, and FPN.
• Across Pascal Context, COCO-stuff, and ADE20K, GFFNet variants achieve top or near-top results, indicating generality across datasets and backbones.