[Paper Review] Improving the Resolution of CNN Feature Maps Efficiently with Multisampling

Shayan Sadigh, Pradeep Sen|arXiv (Cornell University)|May 28, 2018

Advanced Neural Network Applications17 references3 citations

TL;DR

This paper introduces multisampling, a novel technique to enhance CNN feature map resolution by preserving more spatial information during subsampling. Checkered subsampling, a specific implementation, maintains 50% spatial resolution (vs. 25% in traditional max-pooling) using a structured sampling pattern, boosting accuracy in ResNet and DenseNet without additional parameters, and even improving pretrained ImageNet models without fine-tuning.

ABSTRACT

We describe a new class of subsampling techniques for CNNs, termed multisampling, that significantly increases the amount of information kept by feature maps through subsampling layers. One version of our method, which we call checkered subsampling, significantly improves the accuracy of state-of-the-art architectures such as DenseNet and ResNet without any additional parameters and, remarkably, improves the accuracy of certain pretrained ImageNet models without any training or fine-tuning. We glean possible insight into the nature of data augmentations and demonstrate experimentally that coarse feature maps are bottlenecking the performance of neural networks in image classification.

Motivation & Objective

• Address the bottleneck caused by coarse final feature maps in standard CNNs, which limits accuracy in image classification.
• Overcome the trade-off between computational efficiency and spatial information retention in subsampling layers.
• Preserve high-resolution spatial features without increasing model parameters or computational cost.
• Enable state-of-the-art models to achieve higher accuracy through minimal architectural modifications.
• Demonstrate that coarse feature maps are a key performance bottleneck in modern CNNs.

Proposed method

• Propose multisampling as a general framework to increase the number of samples taken from feature maps during subsampling, preserving more spatial information.
• Introduce checkered subsampling, a structured 2D subsampling method using a checkerboard-like pattern that selects 50% of spatial locations per layer instead of 25%.
• Extend feature maps with a submap dimension to store multiple sampled feature representations, enabling 3D convolutional operations across submaps.
• Use low-discrepancy lattice sequences or random sampler selection to avoid diagonal sampling bias when applying checkered subsampling repeatedly.
• Implement the method in deep learning frameworks like PyTorch with minimal code changes, allowing easy conversion of existing architectures like ResNet and DenseNet into checkered CNNs (CCNNs).
• Apply the same subsampling pattern across all submaps or vary it randomly to improve distribution and reduce spatial bias in feature sampling.

Experimental results

Research questions

• RQ1 Can we improve the resolution of CNN feature maps without increasing model parameters or computational cost?
• RQ2 Does preserving more spatial information in subsampling layers lead to higher accuracy in image classification tasks?
• RQ3 Can checkered subsampling be applied to existing pretrained models to improve accuracy without fine-tuning?
• RQ4 Is the performance degradation in modern CNNs due to coarse final feature maps, and can this be mitigated?
• RQ5 How does the structured sampling pattern of checkered subsampling compare to traditional max-pooling and dilated convolutions in terms of information retention and computational efficiency?

Key findings

• Checkered subsampling increases the spatial resolution of feature maps to 50% of the original size per layer, compared to 25% in standard max-pooling.
• The method improves the accuracy of state-of-the-art models like ResNet and DenseNet without adding any parameters.
• Pretrained ImageNet models converted to checkered CNNs show improved top-1 accuracy without any retraining or fine-tuning.
• The technique maintains computational efficiency by reducing spatial dimensions while preserving more spatial information, avoiding the high cost of dilated convolutions.
• Randomly applying different checkered samplers across submaps prevents diagonal sampling bias and improves feature distribution.
• The submap dimension enables 3D convolutions that process multiple sampled feature representations, enhancing feature representation capacity.