[Paper Review] Ordinal Pooling Networks: For Preserving Information over Shrinking Feature Maps

Ashwani Kumar|arXiv (Cornell University)|Apr 8, 2018

Advanced Neural Network Applications19 references7 citations

TL;DR

This paper proposes Ordinal Pooling Networks (OPN), a learnable pooling mechanism that preserves more feature information than traditional max-pooling by ranking and weighting elements within a pooling region according to their ordinal positions. Experiments show OPN improves image classification accuracy over max-pooling, with gains expected to increase in deeper networks due to reduced information loss during downsampling.

ABSTRACT

In the framework of convolutional neural networks that lie at the heart of deep learning, downsampling is often performed with a max-pooling operation that only retains the element with maximum activation, while completely discarding the information contained in other elements in a pooling region. To address this issue, a novel pooling scheme, Ordinal Pooling Network (OPN), is introduced in this work. OPN rearranges all the elements of a pooling region in a sequence and assigns different weights to these elements based upon their orders in the sequence, where the weights are learned via the gradient-based optimisation. The results of our small-scale experiments on image classification task demonstrate that this scheme leads to a consistent improvement in the accuracy over max-pooling operation. This improvement is expected to increase in deeper networks, where several layers of pooling become necessary.

Motivation & Objective

• To address the information loss inherent in max-pooling, which discards all but the highest activation in a pooling region.
• To develop a pooling mechanism that retains more spatial and activation-level information during feature map downsampling.
• To introduce a differentiable, gradient-optimized pooling strategy that assigns learned weights based on the relative order of activations within a pooling window.
• To evaluate the effectiveness of the proposed method on image classification tasks and demonstrate its scalability to deeper networks.

Proposed method

• The method rearranges all elements in a pooling region into a sorted sequence based on their activation values.
• It assigns learnable weights to each position in the sorted sequence, where the weight corresponds to the ordinal rank of the element in the sequence.
• The pooling operation computes a weighted sum of the sorted activations using these learned ordinal weights.
• The weights are optimized end-to-end via backpropagation during network training.
• The approach is applied in a standard CNN architecture, replacing max-pooling layers with OPN layers.
• The method is designed to be compatible with existing CNN frameworks and trainable with standard optimization techniques.

Experimental results

Research questions

• RQ1 Can a pooling mechanism that retains more than just the maximum activation improve classification accuracy in deep CNNs?
• RQ2 Does learning ordinal weights for feature map elements lead to better representation learning compared to fixed max-pooling?
• RQ3 How does the performance of OPN compare to max-pooling on standard image classification benchmarks?
• RQ4 Does the benefit of OPN increase with network depth due to cumulative information preservation across multiple pooling layers?

Key findings

• OPN achieves consistent accuracy improvements over max-pooling in small-scale image classification experiments.
• The proposed method reduces information loss during downsampling by preserving relative activation order and magnitude.
• The improvement in accuracy is expected to grow in deeper networks due to the cumulative effect of reduced pooling-induced information loss.
• The learned ordinal weights adaptively emphasize informative features based on their rank, enabling more robust feature representation.