[Paper Review] Multi Layer Neural Networks as Replacement for Pooling Operations

Wolfgang Fuhl, Enkelejda Kasneci|arXiv (Cornell University)|Jun 12, 2020

Computational Physics and Python Applications64 references6 citations

TL;DR

This paper proposes using single perceptrons or shallow multi-layer neural networks as learnable pooling operations in convolutional neural networks, replacing traditional max or average pooling. By restructuring feature maps into perceptron inputs, the method achieves competitive accuracy with minimal parameter increase and enables end-to-end training, including for upsampling in semantic segmentation tasks.

ABSTRACT

Pooling operations, which can be calculated at low cost and serve as a linear or nonlinear transfer function for data reduction, are found in almost every modern neural network. Countless modern approaches have already tackled replacing the common maximum value selection and mean value operations, not to mention providing a function that allows different functions to be selected through changing parameters. Additional neural networks are used to estimate the parameters of these pooling functions.Consequently, pooling layers may require supplementary parameters to increase the complexity of the whole model. In this work, we show that one perceptron can already be used effectively as a pooling operation without increasing the complexity of the model. This kind of pooling allows for the integration of multi-layer neural networks directly into a model as a pooling operation by restructuring the data and, as a result, learnin complex pooling operations. We compare our approach to tensor convolution with strides as a pooling operation and show that our approach is both effective and reduces complexity. The restructuring of the data in combination with multiple perceptrons allows for our approach to be used for upscaling, which can then be utilized for transposed convolutions in semantic segmentation.

Motivation & Objective

• To address the limitations of fixed pooling operations (e.g., max, average) that lack adaptability to input data and task-specific patterns.
• To reduce model complexity while improving performance by replacing traditional pooling with learnable, parameterized operations.
• To explore whether simple neural networks can serve as effective, low-complexity replacements for pooling layers without increasing computational overhead.
• To enable end-to-end training of pooling operations by integrating them directly into the network architecture without auxiliary branches.
• To extend the approach to upsampling operations using the same neural network-based pooling mechanism for use in transposed convolutions and semantic segmentation.

Proposed method

• Reconfigure feature map activations into a 1D vector for input to a single perceptron or shallow multi-layer network, replacing standard pooling layers.
• Use a perceptron with learnable weights and bias as a differentiable, parameterized pooling function that can be trained end-to-end.
• Structure multi-layer perceptrons (e.g., 4-1 or 4-16-1) to learn complex, non-linear pooling functions from feature maps.
• Apply the same neural network architecture to both downsampling (pooling) and upsampling (transposed convolution) operations via data restructuring.
• Train the entire model end-to-end using backpropagation, with the pooling operation’s parameters updated jointly with other network weights.
• Use strided convolutions as a baseline for comparison, demonstrating that the proposed method achieves similar or better performance with fewer parameters.

Experimental results

Research questions

• RQ1 Can a single perceptron effectively replace traditional pooling operations like max or average pooling in CNNs without increasing model complexity?
• RQ2 How does the performance of a learnable perceptron-based pooling layer compare to fixed pooling and strided convolutional pooling in image classification tasks?
• RQ3 Can multi-layer perceptrons be used as pooling operations to learn complex, non-linear feature reduction patterns?
• RQ4 Can the same neural network-based pooling mechanism be effectively applied to upsampling operations in semantic segmentation architectures?
• RQ5 Does integrating learnable pooling layers into the main network path improve model accuracy while maintaining computational efficiency?

Key findings

• The perceptron-based pooling operation outperformed both max and average pooling in three out of four runs on the CIFAR10 dataset using model c) from Figure 2.
• The NN-4-1 (4 perceptrons in hidden layer, 1 output) achieved the best overall performance among all pooling methods tested on CIFAR10, with results comparable to strided convolutional pooling.
• On the challenging VOC2012 semantic segmentation dataset, replacing standard pooling and upsampling layers with perceptron-based operations improved segmentation accuracy.
• The method achieved competitive performance with strided convolutions while introducing significantly fewer parameters and maintaining O(n) computational complexity.
• The approach is stable under training and can be effectively optimized by reducing the learning rate when instability occurs.
• The same neural network architecture used for pooling can be repurposed for upsampling via data restructuring, enabling effective transposed convolution learning.