[Paper Review] Resolution-Aware Design of Atrous Rates for Semantic Segmentation Networks

Bum Jun Kim, Hyeyeon Choi+2 more|arXiv (Cornell University)|Jul 26, 2023

Advanced Neural Network ApplicationsComputer Science3 citations

TL;DR

This paper proposes resolution-aware guidelines for selecting optimal atrous rates in Semantic Segmentation networks, specifically for the Atrous Spatial Pyramid Pooling (ASPP) module. By analyzing the Effective Receptive Field (ERF) of segmentation networks, the authors identify that ASPP induces a star-shaped ERF pattern and derive a formula to set the optimal atrous rate based on input image size and feature map stride, resulting in consistent mIoU improvements across multiple datasets including Cityscapes, STARE, and iSAID.

ABSTRACT

DeepLab is a widely used deep neural network for semantic segmentation, whose success is attributed to its parallel architecture called atrous spatial pyramid pooling (ASPP). ASPP uses multiple atrous convolutions with different atrous rates to extract both local and global information. However, fixed values of atrous rates are used for the ASPP module, which restricts the size of its field of view. In principle, atrous rate should be a hyperparameter to change the field of view size according to the target task or dataset. However, the manipulation of atrous rate is not governed by any guidelines. This study proposes practical guidelines for obtaining an optimal atrous rate. First, an effective receptive field for semantic segmentation is introduced to analyze the inner behavior of segmentation networks. We observed that the use of ASPP module yielded a specific pattern in the effective receptive field, which was traced to reveal the module's underlying mechanism. Accordingly, we derive practical guidelines for obtaining the optimal atrous rate, which should be controlled based on the size of input image. Compared to other values, using the optimal atrous rate consistently improved the segmentation results across multiple datasets, including the STARE, CHASE_DB1, HRF, Cityscapes, and iSAID datasets.

Motivation & Objective

• To address the limitation of fixed atrous rates (e.g., {6,12,18}) in ASPP modules, which restrict field of view (FoV) size regardless of input resolution.
• To investigate the internal behavior of ASPP modules in semantic segmentation networks through Effective Receptive Field (ERF) analysis.
• To derive practical, resolution-dependent guidelines for selecting optimal atrous rates that maximize FoV alignment with input image size.
• To validate that optimal atrous rates improve segmentation performance across diverse datasets and backbone architectures, including CNNs and vision transformers.

Proposed method

• Introduce a pixel-level Effective Receptive Field (ERF) formulation for semantic segmentation, defined as the gradient contribution of input pixels to a central output unit.
• Visualize ERFs of various segmentation models (e.g., DeepLab, UperNet, SETR) to observe architectural patterns, revealing a star-shaped ERF due to ASPP's parallel atrous convolutions.
• Quantify the FoV size of ASPP by modeling the ERF as a 2D Gaussian and extracting its standard deviations and center coordinates.
• Derive a closed-form expression for the optimal atrous rate $ r^* $ as a function of input resolution $ l $ and output stride $ s $, given by $ r^* = \left\lfloor \frac{l}{2s} \right\rfloor $.
• Evaluate the proposed $ r^* $ across multiple datasets (Cityscapes, STARE, HRF, iSAID, ADE20K) and models (ResNet, ViT, Swin, SegFormer) using mIoU as the metric.
• Use LMfit library to fit ERFs to 2D Gaussians for consistent quantification of ERF width and center bias.

Figure 1 : $3\times 3$ vanilla convolution uses nine adjoint features (shown in blue on the left), whereas $3\times 3$ atrous convolution with an atrous rate of 2 uses nine distant features with one feature in between (shown in blue on the right).

Experimental results

Research questions

• RQ1 How does the ASPP module influence the spatial receptive field structure in semantic segmentation networks?
• RQ2 What is the underlying mechanism causing the star-shaped ERF pattern observed in ASPP-equipped networks?
• RQ3 How can the field of view (FoV) size of the ASPP module be systematically controlled based on input image size and network stride?
• RQ4 Does using a resolution-aware atrous rate lead to consistent performance gains across diverse datasets and backbone architectures?

Key findings

• The ASPP module induces a star-shaped ERF pattern due to the symmetric placement of dilated convolutions with different rates, which creates non-uniform influence across the feature map.
• The effective field of view (FoV) size of the ASPP module is quantitatively determined by the spread of the ERF, which can be modeled as a 2D Gaussian with a bottom-shifted center and wider horizontal extent.
• The optimal atrous rate $ r^* = \left\lfloor \frac{l}{2s} \right\rfloor $, derived from FoV alignment with input resolution, consistently improves mIoU across all evaluated datasets and models.
• On the Cityscapes dataset using SETR with ViT-L/16, the proposed $ r^* = 8 $ achieved a +0.34% mIoU improvement over the baseline, outperforming all other tested rates from 1 to 12.
• Transformer-based models like SegFormer and SETR with ViT backbones exhibit small, localized ERFs despite global self-attention, indicating that local inductive bias remains critical even in attention-based architectures.
• The ERF of ViT-based models shows a patch-partitioned 2D Gaussian pattern, while Swin-based models exhibit a smooth, continuous ERF due to local window attention and feature merging.

(a) DeepLabV3 with R-18, $s=8$