[Paper Review] Very Efficient Training of Convolutional Neural Networks using Fast Fourier Transform and Overlap-and-Add

Tyler Highlander, Andrés Rodríguez|arXiv (Cornell University)|Jan 25, 2016

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes a novel method for accelerating convolutional neural network (CNN) training by combining the Fast Fourier Transform (FFT) with the overlap-and-add (OaA) technique, reducing computational complexity from O(N²n²) to O(N² log₂n) per kernel. The approach achieves up to 16.3× speedup over traditional convolution in forward and backward propagation, especially effective when input size N is much larger than kernel size n.

ABSTRACT

Convolutional neural networks (CNNs) are currently state-of-the-art for various classification tasks, but are computationally expensive. Propagating through the convolutional layers is very slow, as each kernel in each layer must sequentially calculate many dot products for a single forward and backward propagation which equates to $\\mathcal{O}(N^{2}n^{2})$ per kernel per layer where the inputs are $N \ imes N$ arrays and the kernels are $n \ imes n$ arrays. Convolution can be efficiently performed as a Hadamard product in the frequency domain. The bottleneck is the transformation which has a cost of $\\mathcal{O}(N^{2}\\log_2 N)$ using the fast Fourier transform (FFT). However, the increase in efficiency is less significant when $N\\gg n$ as is the case in CNNs. We mitigate this by using the "overlap-and-add" technique reducing the computational complexity to $\\mathcal{O}(N^2\\log_2 n)$ per kernel. This method increases the algorithm's efficiency in both the forward and backward propagation, reducing the training and testing time for CNNs. Our empirical results show our method reduces computational time by a factor of up to 16.3 times the traditional convolution implementation for a 8 $\ imes$ 8 kernel and a 224 $\ imes$ 224 image.

Motivation & Objective

• To reduce the computational bottleneck in training deep CNNs, which is dominated by slow convolution operations in spatial domain.
• To improve efficiency of frequency-domain convolution by minimizing the cost of Fourier transforms, especially when input size N ≫ kernel size n.
• To develop a method that maintains model accuracy while drastically reducing training and inference time through optimized FFT-based convolution.
• To demonstrate that overlap-and-add enables faster convolution than standard FFT-based or spatial convolution, particularly in large-input, small-kernel CNN architectures.

Proposed method

• The method partitions the input feature map into non-overlapping blocks of size n×n, matching the kernel size, to enable localized FFTs.
• Each block is convolved with the kernel using FFT: transform block and kernel to frequency domain, perform element-wise (Hadamard) multiplication, then inverse FFT back to spatial domain.
• The resulting convolutions are overlapped and added together to reconstruct the full output, mimicking standard convolution with reduced transform cost.
• The technique reduces per-kernel complexity from O(N² log₂N) in standard FFT convolution to O(N² log₂n), leveraging smaller FFTs on block-sized data.
• The approach is applied to both forward and backward propagation, including error backpropagation and weight gradient computation.
• The method is implemented on CPU with single-threaded execution for fair comparison, but is designed for GPU acceleration using libraries like cuFFT.

Experimental results

Research questions

• RQ1 Can the overlap-and-add technique reduce the computational complexity of CNN convolution beyond standard FFT-based methods?
• RQ2 Does the proposed OaA-FFT method achieve significant speedups in both forward and backward propagation for typical CNN architectures?
• RQ3 How does the performance of OaA-FFT compare to traditional spatial convolution and standard FFT convolution across varying input and kernel sizes?
• RQ4 What is the impact of input size and kernel size on the relative performance gain of the OaA-FFT method?
• RQ5 Can the OaA-FFT method be effectively scaled in practice, especially when considering transform overhead and parallelization potential?

Key findings

• The OaA-FFT method reduces computational complexity per kernel from O(N² log₂N) in standard FFT convolution to O(N² log₂n), achieving a theoretical speedup factor of log₂N / log₂n.
• Empirical results show up to 16.3× speedup over traditional spatial convolution for a 224×224 input and 8×8 kernel, with consistent gains across different network depths.
• The method outperforms standard FFT convolution in both forward and backward propagation, with the backward pass showing even greater relative improvement due to two convolutions per kernel.
• Speedup increases with the number of kernels, as the fixed overhead of FFTs is amortized over more operations.
• The performance peaks at kernel sizes that are powers of two, aligning with optimal FFT performance, suggesting that zero-padding to nearest power-of-two can further enhance efficiency.
• For input sizes larger than 8×8, OaA-FFT consistently outperforms both spatial and standard FFT convolution, especially when N ≫ n.