[Paper Review] ExFuse: Enhancing Feature Fusion for Semantic Segmentation

Zhenli Zhang, Xiangyu Zhang+3 more|arXiv (Cornell University)|Apr 11, 2018

Advanced Neural Network ApplicationsComputer Science31 references54 citations

TL;DR

ExFuse bridges the semantic and resolution gap between low-level and high-level features to enhance feature fusion, yielding a 4% overall gain and state-of-the-art results on PASCAL VOC 2012.

ABSTRACT

Modern semantic segmentation frameworks usually combine low-level and high-level features from pre-trained backbone convolutional models to boost performance. In this paper, we first point out that a simple fusion of low-level and high-level features could be less effective because of the gap in semantic levels and spatial resolution. We find that introducing semantic information into low-level features and high-resolution details into high-level features is more effective for the later fusion. Based on this observation, we propose a new framework, named ExFuse, to bridge the gap between low-level and high-level features thus significantly improve the segmentation quality by 4.0\% in total. Furthermore, we evaluate our approach on the challenging PASCAL VOC 2012 segmentation benchmark and achieve 87.9\% mean IoU, which outperforms the previous state-of-the-art results.

Motivation & Objective

• Motivate and analyze why naive fusion of low-level and high-level features is ineffective due to semantic and resolution gaps.
• Develop techniques to inject semantic information into low-level features and spatial information into high-level features.
• Propose ExFuse as a framework to bridge these gaps and improve segmentation performance.
• Demonstrate improvements across ablations and establish state-of-the-art results on PASCAL VOC 2012.

Proposed method

• Layer rearrangement to make low-level features closer to semantic supervision without changing overall capacity.
• Semantic supervision by attaching auxiliary semantic branches to early encoder stages to enrich low-level features.
• Semantic embedding branch (SEB) to fuse high-level semantic guidance into the residual fusion.
• Explicit channel resolution embedding (ECRE) using a parameter-free upsampling (sub-pixel) to embed higher resolution information into high-level features.
• Densely adjacent prediction (DAP) to propagate spatial information across neighboring pixels by grouping channels for multi-position predictions.

Experimental results

Research questions

• RQ1 Can bridging semantic and resolution gaps between low- and high-level features improve fusion effectiveness in U-Net-like semantic segmentation architectures?
• RQ2 How much do low-level semantic enrichment and high-level spatial enrichment independently contribute to performance gains?
• RQ3 Do the proposed ExFuse techniques generalize beyond GCN to vanilla U-Net and other vision tasks?
• RQ4 What is the impact of using larger backbones and COCO pretraining on VOC 2012 results?
• RQ5 Is the improvement primarily due to the fusion strategy or backbone enhancements?

Key findings

• ExFuse achieves a total 4.0% mIoU gain on PASCAL VOC 2012 validation over the baseline GCN setup.
• On VOC 2012 test, ExFuse with ResNeXt-131 reaches 87.9% mIoU, surpassing prior state-of-the-art methods without post-processing.
• Layer rearrangement, semantic supervision, and SEB each contribute incremental gains to low-level feature quality.
• ECRE with sub-pixel upsampling yields a 0.5% mIoU improvement, demonstrating explicit high-resolution embedding is beneficial.
• DAP provides a 0.6% mIoU gain by embedding spatial information into high-level feature channels.
• With COCO pretraining and test-time augmentation (flips), VOC 2012 validation improves to 85.8% and test to 87.9% mIoU for ExFuse-131, compared to baselines.