[Paper Review] Making Convolutional Networks Shift-Invariant Again

Richard Zhang|arXiv (Cornell University)|Apr 25, 2019

Graph Theory and AlgorithmsComputer Science376 citations

TL;DR

The paper introduces blurring-based downsampling (MaxBlurPool/MaxBlurDown) to make CNNs more shift-invariant, yielding smoother feature maps, reduced sensitivity to spatial shifts, and improved robustness to shift-based adversaries.

ABSTRACT

Modern convolutional networks are not shift-invariant, as small input shifts or translations can cause drastic changes in the output. Commonly used downsampling methods, such as max-pooling, strided-convolution, and average-pooling, ignore the sampling theorem. The well-known signal processing fix is anti-aliasing by low-pass filtering before downsampling. However, simply inserting this module into deep networks degrades performance; as a result, it is seldomly used today. We show that when integrated correctly, it is compatible with existing architectural components, such as max-pooling and strided-convolution. We observe extit{increased accuracy} in ImageNet classification, across several commonly-used architectures, such as ResNet, DenseNet, and MobileNet, indicating effective regularization. Furthermore, we observe extit{better generalization}, in terms of stability and robustness to input corruptions. Our results demonstrate that this classical signal processing technique has been undeservingly overlooked in modern deep networks. Code and anti-aliased versions of popular networks are available at https://richzhang.github.io/antialiased-cnns/ .

Motivation & Objective

• Motivate the need for shift-invariance in convolutional networks and identify limitations of standard pooling.
• Analyze how a blur-based downsampling layer affects the learned convolutional filters.
• Propose and evaluate a MaxBlurPool/MaxBlurDown approach with varying blur kernels.
• Assess the impact on shift-induced performance degradation and adversarial robustness.

Proposed method

• Introduce max-blur based downsampling to encourage smoother feature maps.
• Use Binomial-like blur kernels of varying sizes (e.g., Bin-4, Bin-5, Bin-7) before pooling/downsampling.
• Quantify filter smoothness with normalized Total Variation (TV) across layers.
• Evaluate classification accuracy under spatial shifts and with/without data augmentation.
• Test robustness to shift-based adversaries by measuring performance across shifted input windows.

Experimental results

Research questions

• RQ1 How do the learned convolutional filters change when using the proposed blur-based downsampling?
• RQ2 Does training with this layer produce smoother feature extractors and reduce sensitivity to spatial shifts?
• RQ3 How does accuracy degrade as a function of spatial shift compared to baseline pooling?
• RQ4 Does the approach improve robustness to shift-based adversarial attacks?

Key findings

• MaxBlurPool/Down induces smoother learned filters (lower total variation) than baseline MaxPool.
• Classification accuracy degrades less with spatial shifts when using stronger blur filters (e.g., Bin-7).
• Output class probability variation across shifts decreases with larger blur sizes, reducing sensitivity to input position.
• Stronger blur filtering increases robustness to shift-based adversaries, sometimes outperforming augmented baselines.
• Results corroborate that blur-based downsampling maintains accuracy while improving shift-invariance and robustness.