[Paper Review] Global Aggregation then Local Distribution in Fully Convolutional Networks

Xiangtai Li, Li Zhang+4 more|arXiv (Cornell University)|Sep 16, 2019

Advanced Neural Network Applications40 references34 citations

TL;DR

GALD combines Global Aggregation with a Local Distribution module to adaptively distribute global context per position, improving semantic segmentation, object detection, and instance segmentation across benchmarks.

ABSTRACT

It has been widely proven that modelling long-range dependencies in fully convolutional networks (FCNs) via global aggregation modules is critical for complex scene understanding tasks such as semantic segmentation and object detection. However, global aggregation is often dominated by features of large patterns and tends to oversmooth regions that contain small patterns (e.g., boundaries and small objects). To resolve this problem, we propose to first use \emph{Global Aggregation} and then \emph{Local Distribution}, which is called GALD, where long-range dependencies are more confidently used inside large pattern regions and vice versa. The size of each pattern at each position is estimated in the network as a per-channel mask map. GALD is end-to-end trainable and can be easily plugged into existing FCNs with various global aggregation modules for a wide range of vision tasks, and consistently improves the performance of state-of-the-art object detection and instance segmentation approaches. In particular, GALD used in semantic segmentation achieves new state-of-the-art performance on Cityscapes test set with mIoU 83.3\%. Code is available at: \url{https://github.com/lxtGH/GALD-Net}

Motivation & Objective

• Motivate addressing oversmoothing of small patterns when using global aggregation in FCNs.
• Propose GALD to adaptively distribute global features via per-channel mask maps.
• Demonstrate GALD’s compatibility with existing GA modules and backbones across tasks.
• Show improved performance on semantic segmentation, object detection, and instance segmentation.

Proposed method

• Use a Global Aggregation (GA) module to compute a coarse global feature F_GA from the backbone feature F.
• Introduce Local Distribution (LD) which learns per-channel mask maps M via depth-wise convolutions on F_GA, producing refined features F_GALD = M ⊙ F_GA + F_GA.
• Concatenate F_GALD with original F to form the final feature F_o for task heads (e.g., segmentation, detection).
• LD’s masks M are generated as M = σ(upsample(W_d F_GA)) where σ is a sigmoid and W_d are depth-wise filters.
• GA module can be any existing GA design (e.g., CGNL, NL, PSP, ASPP) and LD is universally compatible.
• Training can employ standard objectives with optional Online Hard Example Mining (OHEM) for segmentation.

Experimental results

Research questions

• RQ1 How does adaptive local distribution of global context affect small vs. large pattern regions in FCNs?
• RQ2 Can GALD consistently improve performance when attached to various GA modules and backbones across segmentation, detection, and instance segmentation tasks?
• RQ3 What is the impact of different mask-downsampling strategies on the LD module’s effectiveness?
• RQ4 Does GALD enable state-of-the-art single-model performance on Cityscapes with standard backbones?
• RQ5 How does GALD perform with additional data sources (e.g., Mapillary) for Cityscapes?

Key findings

• GALD consistently improves GA modules (e.g., CGNL, NL, PSP, ASPP) when attached to FCN backbones for semantic segmentation.
• On Cityscapes semantic segmentation, a single-model ResNet101 GALD setup achieves 83.3% mIoU on the test set (with Mapillary data).
• LD alone improves FCN by 3.8% mIoU; GALD (GA + LD) yields the best gains across configurations.
• For Pascal VOC detection and COCO instance segmentation, GALD yields approximately 0.5–1.0% gains in AP-box and AP-mask over strong baselines.
• Ablation results show depth-wise downsampling for mask estimation in LD performs best among tested strategies, with CGNL+LD achieving the strongest results in several setups.
• Compared to baseline Faster R-CNN and Mask R-CNN with ResNet backbones, GALD improves mAP@0.5 and AP-mask, demonstrating generalization across tasks and datasets.