[Paper Review] How Much Position Information Do Convolutional Neural Networks Encode?

Amirul Islam, Sen Jia+1 more|arXiv (Cornell University)|Jan 22, 2020

Advanced Neural Network ApplicationsComputer Science25 references93 citations

TL;DR

The paper demonstrates that absolute position information is implicitly encoded in CNN feature maps, even when networks are trained only for object recognition, and analyzes how padding and network depth contribute to this positional encoding.

ABSTRACT

In contrast to fully connected networks, Convolutional Neural Networks (CNNs) achieve efficiency by learning weights associated with local filters with a finite spatial extent. An implication of this is that a filter may know what it is looking at, but not where it is positioned in the image. Information concerning absolute position is inherently useful, and it is reasonable to assume that deep CNNs may implicitly learn to encode this information if there is a means to do so. In this paper, we test this hypothesis revealing the surprising degree of absolute position information that is encoded in commonly used neural networks. A comprehensive set of experiments show the validity of this hypothesis and shed light on how and where this information is represented while offering clues to where positional information is derived from in deep CNNs.

Motivation & Objective

• Investigate whether CNNs encode absolute position information in learned feature maps.
• Quantify position information in pretrained CNNs using synthetic ground-truth position maps.
• Analyze how padding and network architecture affect the encoding of position information.
• Determine which layers in CNNs carry more positional information and how readout of this information can be improved.

Proposed method

• Freeze encoder networks (e.g., VGG16, ResNet) pretrained on ImageNet.
• Attach a lightweight position encoding module (PosENet) that reads multi-scale features and outputs a gradient-like position map hat(fp).
• Train the PosENet with pixel-wise mean squared error against synthetic ground-truth position maps (H: horizontal gradient, V: vertical gradient, G: Gaussian, HS/VS: stripes).
• Evaluate position-readout capability via Spearman correlation (SPC) and mean absolute error (MAE) on natural and synthetic datasets.
• Conduct ablations on layers, kernel sizes, and padding to understand how position information is stored and how it is read out.

Experimental results

Research questions

• RQ1 Do CNNs trained for object recognition implicitly encode absolute positional information in their feature maps?
• RQ2 How easily can a readout network extract absolute position information from pretrained CNN features?
• RQ3 What roles do network depth, kernel size, and padding play in encoding and extracting position information?
• RQ4 Which layers in pretrained CNNs carry more positional information, and how does semantic content affect this encoding?

Key findings

• Position information is robustly encoded in commonly used CNNs, and can be extracted by a lightweight readout (PosENet) trained on top of frozen encoders.
• ResNet-based readouts outperform VGG-based ones in recovering absolute position, indicating deeper/seemingly more semantic representations help encode position.
• Larger receptive fields (bigger kernel sizes) and deeper readouts improve the extraction of position information.
• Zero-padding at borders is a key source of position information; removing padding reduces, while adding padding enhances, positional readout performance.
• Position information is stronger in deeper encoder features (fpos5) than in shallow layers (fpos1), suggesting high-level features carry more spatial cues.
• When tasks involve semantics (saliency, segmentation), zero-padding continues to influence positional cues and may be reinforced by pretraining on non-classification tasks.