[Paper Review] Hartley Spectral Pooling for Deep Learning

Hao Zhang, Jianwei Ma|arXiv (Cornell University)|Oct 7, 2018

Neural Networks and Applications35 references16 citations

TL;DR

This paper proposes Hartley Spectral Pooling (HSP), a real-valued spectral pooling method that replaces traditional max/average pooling or strided convolutions in CNNs by transforming feature maps into the Hartley frequency domain, truncating high frequencies, and transforming back. HSP preserves more structural information than conventional pooling, leading to faster convergence and improved accuracy on MNIST, Fashion-MNIST, and CIFAR-10, with a 0.32% test error on MNIST and 8.63% on CIFAR-10, while avoiding complex arithmetic used in Fourier-based methods.

ABSTRACT

In most convolution neural networks (CNNs), downsampling hidden layers is adopted for increasing computation efficiency and the receptive field size. Such operation is commonly so-called pooling. Maximation and averaging over sliding windows (max/average pooling), and plain downsampling in the form of strided convolution are popular pooling methods. Since the pooling is a lossy procedure, a motivation of our work is to design a new pooling approach for less lossy in the dimensionality reduction. Inspired by the Fourier spectral pooling(FSP) proposed by Rippel et. al. [1], we present the Hartley transform based spectral pooling method in CNNs. Compared with FSP, the proposed spectral pooling avoids the use of complex arithmetic for frequency representation and reduces the computation. Spectral pooling preserves more structure features for network's discriminability than max and average pooling. We empirically show that Hartley spectral pooling gives rise to the convergence of training CNNs on MNIST and CIFAR-10 datasets.

Motivation & Objective

• To design a less lossy dimensionality reduction method for CNNs that preserves more structural information than max or average pooling.
• To overcome the limitations of complex-valued spectral pooling (e.g., Fourier-based), which requires handling imaginary components and conjugate symmetry.
• To propose a real-valued alternative using the Hartley transform that maintains computational efficiency and compatibility with standard CNNs.
• To investigate whether preserving more frequency-domain information improves training convergence and generalization in deep networks.
• To compare HSP’s performance and efficiency against Fourier-based and DCT-based spectral pooling, as well as standard max/average pooling and strided convolutions.

Proposed method

• The method applies the Discrete Hartley Transform (DHT) to each feature map to convert spatial data into real-valued frequency components.
• It performs low-pass filtering by truncating the highest-frequency components of the DHT spectrum, retaining only the lowest-frequencies.
• The truncated spectrum is then inverse-transformed back to the spatial domain to produce a downsampled feature map with reduced dimensionality.
• The approach avoids complex arithmetic entirely, as the Hartley transform operates purely on real numbers, simplifying implementation and reducing computational overhead.
• The method is integrated into residual networks (ResNet) by replacing strided convolutions or pooling layers with HSP layers at specific downsampling stages.
• The output dimension of each HSP layer is configurable (e.g., 20×20, 12×12, 4×4), allowing flexible control over spatial reduction.

Experimental results

Research questions

• RQ1 Does spectral pooling using the Hartley transform lead to better feature preservation and classification accuracy than max or average pooling in CNNs?
• RQ2 Can Hartley spectral pooling improve training convergence speed and stability in deep networks compared to standard pooling or strided convolutions?
• RQ3 How does the performance of Hartley spectral pooling compare to Fourier-based and DCT-based spectral pooling in terms of accuracy, training time, and parameter efficiency?
• RQ4 Does the use of real-valued transforms in spectral pooling eliminate the need for complex arithmetic and conjugate symmetry corrections, simplifying implementation?
• RQ5 Can HSP be effectively integrated into modern residual networks without architectural overhaul, and does it maintain or improve generalization?

Key findings

• Hartley Spectral Pooling (HSP) achieved a test error of 0.32% on MNIST, outperforming ResNet20 (0.36%) and ResNet16 (0.36%), with a 15% parameter reduction.
• On Fashion-MNIST, HSP reduced test error to 6.26% (best), compared to 6.91% for ResNet20, with a 30% increase in training time per epoch.
• On CIFAR-10, HSP achieved a test error of 8.63%, outperforming ResNet16 (8.87%) and DCTSP-ResNet15 (8.81%), though with a 1000s longer training time.
• HSP led to faster convergence in early training stages, as shown in training curves, indicating improved optimization dynamics.
• Despite slightly lower accuracy than DCTSP, HSP was significantly faster, with training time roughly half that of DCTSP on Fashion-MNIST and less than half on CIFAR-10.
• The method demonstrated that preserving more frequency-domain information enhances discriminability and supports better generalization in deep networks.