[Paper Review] Revisiting Multi-Scale Feature Fusion for Semantic Segmentation

Tianjian Meng, Golnaz Ghiasi+3 more|arXiv (Cornell University)|Mar 23, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes ESeg, a simplified semantic segmentation model that achieves state-of-the-art accuracy without relying on high internal resolution or expensive atrous convolutions. By extending the multi-scale feature space up to $P_9$ and using a bidirectional BiFPN for feature fusion, ESeg achieves 76.0% mIoU at 189 FPS on CityScapes—surpassing FasterSeg in both speed and accuracy—demonstrating that efficient, high-performance segmentation is possible with a streamlined architecture.

ABSTRACT

It is commonly believed that high internal resolution combined with expensive operations (e.g. atrous convolutions) are necessary for accurate semantic segmentation, resulting in slow speed and large memory usage. In this paper, we question this belief and demonstrate that neither high internal resolution nor atrous convolutions are necessary. Our intuition is that although segmentation is a dense per-pixel prediction task, the semantics of each pixel often depend on both nearby neighbors and far-away context; therefore, a more powerful multi-scale feature fusion network plays a critical role. Following this intuition, we revisit the conventional multi-scale feature space (typically capped at P5) and extend it to a much richer space, up to P9, where the smallest features are only 1/512 of the input size and thus have very large receptive fields. To process such a rich feature space, we leverage the recent BiFPN to fuse the multi-scale features. Based on these insights, we develop a simplified segmentation model, named ESeg, which has neither high internal resolution nor expensive atrous convolutions. Perhaps surprisingly, our simple method can achieve better accuracy with faster speed than prior art across multiple datasets. In real-time settings, ESeg-Lite-S achieves 76.0% mIoU on CityScapes [12] at 189 FPS, outperforming FasterSeg [9] (73.1% mIoU at 170 FPS). Our ESeg-Lite-L runs at 79 FPS and achieves 80.1% mIoU, largely closing the gap between real-time and high-performance segmentation models.

Motivation & Objective

• To challenge the prevailing belief that high internal resolution and atrous convolutions are essential for accurate semantic segmentation.
• To investigate whether a richer multi-scale feature space and improved feature fusion can replace expensive components like atrous convolutions.
• To develop a simpler, faster, and more efficient segmentation model that maintains or exceeds SOTA performance.
• To close the performance gap between real-time and high-accuracy segmentation models.

Proposed method

• Extends the conventional multi-scale feature space from $P_5$ to $P_9$, where the smallest features are $1/512$ the input size, enabling large receptive fields.
• Uses a lightweight encoder-decoder structure with an off-the-shelf EfficientNet as the encoder and a modified BiFPN as the decoder.
• Employs BiFPN for bidirectional feature fusion, combining top-down and bottom-up pathways to enhance feature integration across scales.
• Avoids atrous convolutions and high-resolution feature maps entirely, reducing computational cost and memory usage.
• Leverages self-training and pretraining on large-scale datasets like Mapillary Vistas to boost performance on CityScapes.
• Optimizes for inference efficiency by minimizing FLOPs and model size while maximizing mIoU.

Experimental results

Research questions

• RQ1 Can semantic segmentation achieve SOTA accuracy without relying on high internal resolution or atrous convolutions?
• RQ2 How does extending the multi-scale feature space to $P_9$ impact feature representation and segmentation accuracy?
• RQ3 Can a bidirectional feature fusion mechanism like BiFPN outperform conventional top-down or skip connections in a rich multi-scale setting?
• RQ4 To what extent can a simple encoder-decoder architecture with minimal components match or exceed complex SOTA models?
• RQ5 Can self-training and pretraining on large datasets further close the gap between real-time and high-performance segmentation models?

Key findings

• ESeg-Lite-S achieves 76.0% mIoU on CityScapes at 189 FPS, outperforming FasterSeg (73.1% mIoU at 170 FPS) in both accuracy and speed.
• ESeg-Lite-L achieves 80.1% mIoU at 79 FPS, significantly narrowing the performance gap between real-time and high-accuracy models.
• The ESeg model uses 2× to 9× fewer parameters and 4× to 50× fewer FLOPs than competitive methods while matching or exceeding their accuracy.
• With pretraining on Mapillary Vistas and self-training on CityScapes coarse data, ESeg-L achieves 84.8% mIoU on CityScapes, setting a new single-model, single-scale SOTA.
• BiFPN outperforms DeepLabV3+ with or without atrous convolutions, proving that feature fusion architecture is more critical than costly modules.
• The ablation study confirms that high internal resolution is expensive and unnecessary when combined with a powerful multi-scale fusion mechanism.