[Paper Review] Spectral Representations for Convolutional Neural Networks

Oren Rippel, Jasper Snoek+1 more|arXiv (Cornell University)|Jun 11, 2015

Sparse and Compressive Sensing TechniquesEngineering19 references186 citations

TL;DR

This paper introduces spectral representations for convolutional neural networks (CNNs) by leveraging the discrete Fourier transform (DFT) to enable spectral pooling and complex-coefficient filter parametrization. Spectral pooling reduces dimensionality by truncating frequency components, preserving more information than max pooling while allowing flexible output sizes. Complex spectral parametrization accelerates training convergence by 2–5× by exploiting sparsity in the frequency domain, achieving competitive performance without dropout or max-pooling.

ABSTRACT

Discrete Fourier transforms provide a significant speedup in the computation of convolutions in deep learning. In this work, we demonstrate that, beyond its advantages for efficient computation, the spectral domain also provides a powerful representation in which to model and train convolutional neural networks (CNNs). We employ spectral representations to introduce a number of innovations to CNN design. First, we propose spectral pooling, which performs dimensionality reduction by truncating the representation in the frequency domain. This approach preserves considerably more information per parameter than other pooling strategies and enables flexibility in the choice of pooling output dimensionality. This representation also enables a new form of stochastic regularization by randomized modification of resolution. We show that these methods achieve competitive results on classification and approximation tasks, without using any dropout or max-pooling. Finally, we demonstrate the effectiveness of complex-coefficient spectral parameterization of convolutional filters. While this leaves the underlying model unchanged, it results in a representation that greatly facilitates optimization. We observe on a variety of popular CNN configurations that this leads to significantly faster convergence during training.

Motivation & Objective

• To explore the frequency domain as a powerful representation for modeling and training CNNs beyond computational efficiency.
• To address the information loss and inflexible dimensionality reduction inherent in traditional pooling methods like max pooling.
• To improve optimization efficiency in CNN training by reparametrizing filters in the spectral domain.
• To enable flexible, resolution-controlled dimensionality reduction through frequency truncation.
• To demonstrate that spectral representations can achieve competitive performance without relying on dropout or max-pooling.

Proposed method

• Proposes spectral pooling: dimensionality reduction via truncation of the frequency representation after applying the DFT to feature maps.
• Introduces complex-coefficient spectral parametrization of convolutional filters, where filters are learned directly in the frequency domain and transformed back to the spatial domain via inverse DFT.
• Utilizes the unitary property of the DFT to ensure the model capacity remains unchanged under spectral reparametrization.
• Employs randomized resolution modification as a form of stochastic regularization by stochastically truncating frequency components during training.
• Applies the Adam optimizer to compare convergence speed between spatial and spectral filter parametrizations.
• Implements spectral pooling at negligible computational cost in networks already using FFT-based convolutions, requiring only matrix truncation.

Experimental results

Research questions

• RQ1 Can the frequency domain provide a more informative and efficient representation for dimensionality reduction in CNNs compared to traditional pooling?
• RQ2 Does learning filters in the spectral domain improve optimization dynamics and convergence speed in CNN training?
• RQ3 To what extent can spectral pooling preserve information while enabling arbitrary output dimensionality?
• RQ4 Can spectral representations reduce or eliminate the need for dropout and max-pooling in CNNs?
• RQ5 How does spectral parametrization affect the optimization landscape and convergence behavior of CNNs with varying filter sizes and architectures?

Key findings

• Spectral pooling preserves significantly more information per parameter than max pooling, especially due to the concentration of signal energy in low frequencies.
• Spectral pooling enables flexible, non-uniform dimensionality reduction by allowing any desired output size through controlled truncation of frequency components.
• Complex-coefficient spectral parametrization leads to 2–5× faster convergence during training across multiple CNN architectures and filter sizes.
• The method achieves competitive classification and approximation performance on benchmark tasks without using dropout or max-pooling.
• Spectral representations exploit sparsity in filter frequency components, providing more meaningful optimization directions than spatial-domain representations.
• The approach is compatible with batch normalization and maintains efficiency when used in conjunction with FFT-based convolutions.