[Paper Review] Partial Convolution based Padding

Guilin Liu, Kevin J. Shih+6 more|arXiv (Cornell University)|Nov 28, 2018

Advanced Neural Network Applications2 references75 citations

TL;DR

The paper proposes a drop-in padding scheme called partial convolution based padding that treats padded areas as holes and re-weights convolution outputs to account for missing data, improving ImageNet accuracy and border-region segmentation stability.

ABSTRACT

In this paper, we present a simple yet effective padding scheme that can be used as a drop-in module for existing convolutional neural networks. We call it partial convolution based padding, with the intuition that the padded region can be treated as holes and the original input as non-holes. Specifically, during the convolution operation, the convolution results are re-weighted near image borders based on the ratios between the padded area and the convolution sliding window area. Extensive experiments with various deep network models on ImageNet classification and semantic segmentation demonstrate that the proposed padding scheme consistently outperforms standard zero padding with better accuracy.

Motivation & Objective

• Motivate and address shortcomings of standard padding schemes (zero, reflection, replication) at image borders.
• Propose a padding method that conditions convolution output only on valid inputs by leveraging partial convolution concepts.
• Show that the proposed padding improves accuracy and convergence across multiple CNN architectures on ImageNet.
• Demonstrate robustness to padding type during inference and improvements in border-region segmentation tasks.

Proposed method

• Define padding as border holes and apply partial convolution re-weighting to adjust the convolution output by the ratio of valid to total input within the sliding window.
• Use a re-weighting factor r(i,j) = ||1_p1(i,j)||1 / ||M_p0(i,j)||1 to scale the convolution when padding introduces missing data.
• Update the mask after each convolution to mark positions conditioned on at least one valid input.
• Explain that zero padding is a special case of partial convolution based padding when r(i,j) = 1.
• Provide a pure-PyTorch implementation that caches the re-weighting factor after the first iteration for efficiency.
• Demonstrate how to extend partial convolution to large padding sizes by propagating and updating the mask through layers.

Experimental results

Research questions

• RQ1 Does replacing zero padding with partial convolution based padding improve accuracy across CNN architectures on ImageNet and CityScapes?
• RQ2 Is partial convolution based padding robust to changes in padding type during inference?
• RQ3 How does border-focused re-weighting affect segmentation performance near image boundaries?
• RQ4 What are the convergence and stability implications when using partial convolution based padding compared to standard padding schemes?

Key findings

• Partial convolution based padding consistently improves validation accuracy over zero padding across VGG and ResNet families on ImageNet.
• The improvement is notable for VGG19 and ResNet50 (≈0.32–0.68% top-1 gain in reported runs).
• Partial convolution based padding yields smaller performance variance (lower standard deviation) than zero padding, indicating greater robustness.
• Models with partial convolution based padding converge faster to comparable or higher accuracy than their zero-padded counterparts.
• In semantic segmentation (DeepLabV3+ on Cityscapes), partial convolution based padding improves mIOU for both ResNet50 and WideResNet38 backbones, with tile-based evaluation showing additional gains over regular evaluation, especially for border regions.