[Paper Review] Position, Padding and Predictions: A Deeper Look at Position Information in CNNs

Md Amirul Islam, Matthew Kowal+3 more|arXiv (Cornell University)|Jan 28, 2021

Advanced Neural Network ApplicationsComputer Science71 references41 citations

TL;DR

The paper investigates how absolute position information is encoded in CNNs, showing zero padding drives position encoding, analyzes border effects, and demonstrates how padding and canvas choices influence downstream tasks like classification and segmentation.

ABSTRACT

In contrast to fully connected networks, Convolutional Neural Networks (CNNs) achieve efficiency by learning weights associated with local filters with a finite spatial extent. An implication of this is that a filter may know what it is looking at, but not where it is positioned in the image. In this paper, we first test this hypothesis and reveal that a surprising degree of absolute position information is encoded in commonly used CNNs. We show that zero padding drives CNNs to encode position information in their internal representations, while a lack of padding precludes position encoding. This gives rise to deeper questions about the role of position information in CNNs: (i) What boundary heuristics enable optimal position encoding for downstream tasks?; (ii) Does position encoding affect the learning of semantic representations?; (iii) Does position encoding always improve performance? To provide answers, we perform the largest case study to date on the role that padding and border heuristics play in CNNs. We design novel tasks which allow us to quantify boundary effects as a function of the distance to the border. Numerous semantic objectives reveal the effect of the border on semantic representations. Finally, we demonstrate the implications of these findings on multiple real-world tasks to show that position information can both help or hurt performance.

Motivation & Objective

• Assess whether CNNs encode absolute position information and identify the padding types that maximize this encoding.
• Examine how boundary effects influence learned representations and downstream task performance.
• Understand how canvas/background choices interact with padding to shape position encoding and semantic features.
• Provide per-location analysis to reveal how border proximity affects learning and decisions.

Proposed method

• Introduce Position Encoding Network (PosENet) that pairs a convolutional encoder with a simple position encoding module to predict absolute position maps.
• Generate gradient-like and Gaussian ground-truth maps to quantify position information captured by different CNN backbones.
• Compare padding types (zero, reflection, replicate, circular) and analyze their effect on position encoding and downstream task performance.
• Conduct location-dependent experiments by placing image patches on canvases of different colors and grid sizes to study boundary effects across image locations.
• Evaluate location-dependent image classification and segmentation using grids of varying sizes (k in {3,5,7,9,11,13}) and analyze robustness to boundary effects.
• Explore no-padding implementations and bilinear interpolation to maintain fair spatial resolution across padding configurations.

Experimental results

Research questions

• RQ1 Does zero padding maximize absolute position information in CNNs compared to other padding types?
• RQ2 How do boundary (border) effects interact with absolute position information across image locations?
• RQ3 Can absolute position information improve or hinder semantic learning depending on the task (e.g., classification vs. segmentation)?
• RQ4 How do canvas color and grid-based placement influence position encoding and downstream performance?
• RQ5 Where in a CNN’s hierarchy is position information stored and how does padding affect this distribution?

Key findings

• Zero padding encodes maximal absolute position information relative to other padding types.
• Position information is more strongly encoded in deeper layers of CNNs, with top layers (f5 in VGG16) showing greater position encoding.
• Padding near borders is the primary source of absolute position signals in pretrained models, and removing padding reduces this information.
• Among common padding types, zero padding yields the strongest position information; circular padding is often the second best, while reflection and replicate provide weaker signals.
• Canvas color (black vs. white vs. mean) interacts with padding to influence boundary effects and task performance, with black canvases often enhancing robustness in no-padding setups.
• Location-dependent experiments reveal border effects impact performance across the input, not just at the border, and padding can modulate this effect.
• Position information can act as a useful feature or a bug depending on the task, aiding position-sensitive tasks like semantic segmentation but potentially hurting translation-invariant tasks like texture recognition.