[Paper Review] Fast Training of Convolutional Networks through FFTs

Michaël Mathieu, Mikael Henaff+1 more|arXiv (Cornell University)|Dec 20, 2013

Advanced Neural Network Applications7 references302 citations

TL;DR

This paper proposes an FFT-based algorithm that accelerates training and inference in convolutional neural networks by computing convolutions as pointwise products in the Fourier domain, reusing transformed feature maps across multiple operations. The method achieves speedups of over an order of magnitude compared to state-of-the-art implementations, particularly for gradient computation and large kernels.

ABSTRACT

Convolutional networks are one of the most widely employed architectures in computer vision and machine learning. In order to leverage their ability to learn complex functions, large amounts of data are required for training. Training a large convolutional network to produce state-of-the-art results can take weeks, even when using modern GPUs. Producing labels using a trained network can also be costly when dealing with web-scale datasets. In this work, we present a simple algorithm which accelerates training and inference by a significant factor, and can yield improvements of over an order of magnitude compared to existing state-of-the-art implementations. This is done by computing convolutions as pointwise products in the Fourier domain while reusing the same transformed feature map many times. The algorithm is implemented on a GPU architecture and addresses a number of related challenges.

Motivation & Objective

• To address the high computational cost of training large convolutional networks, which can take weeks even on modern GPUs.
• To reduce inference time for large-scale datasets, such as web-scale image labeling, where label prediction is computationally expensive.
• To develop a general-purpose algorithm that accelerates all major operations in backpropagation: forward pass, gradient computation, and weight update.
• To explore the feasibility of using FFTs for convolutions in deep networks, especially when the number of feature maps is large.
• To overcome the performance bottleneck of large kernel convolutions by making computation independent of kernel size through FFT-based methods.

Proposed method

• The method leverages the Convolution Theorem, which states that spatial-domain convolutions are equivalent to pointwise products in the Fourier domain: $ f * g = \mathcal{F}^{-1}(\mathcal{F}(f) \cdot \mathcal{F}(g)) $.
• It computes the Fourier transforms of input feature maps and kernels once per layer, then reuses these transforms across all forward, backward, and weight update operations.
• All convolution operations—forward pass, gradient with respect to inputs, and gradient with respect to weights—are reformulated as pointwise multiplications in the frequency domain, avoiding repeated spatial convolutions.
• The algorithm is implemented on GPU architecture, with input and kernel sizes padded to the next power of two to enable efficient FFT computation.
• The method handles circular convolution artifacts by cropping the output to match the size of standard convolutions, ensuring correctness without additional cost.
• The approach is agnostic to kernel size, as kernels are padded to match input size before FFT, making performance independent of kernel dimensions.

Experimental results

Research questions

• RQ1 Can FFT-based convolution significantly accelerate training and inference in deep convolutional networks?
• RQ2 Does reusing precomputed Fourier transforms of feature maps and kernels reduce computational cost across all backpropagation operations?
• RQ3 How does the FFT-based method compare in performance to state-of-the-art GPU-optimized implementations across varying input sizes, kernel sizes, and batch sizes?
• RQ4 Can the method achieve substantial speedups even for large kernels, where direct convolution is most expensive?
• RQ5 Is the performance of the FFT method invariant to kernel size, and can this enable new architectural choices like larger kernels?

Key findings

• The FFT-based method achieves a speedup of over an order of magnitude on the most expensive operation, accGradParameters, compared to state-of-the-art implementations.
• For a typical network configuration (7×7 kernel, 32×32 input, 96 input and 256 output feature maps), the FFT method reduced total training iteration time from 2495 ms (CudaConv) to 867 ms.
• The method outperforms both custom Torch7 and CudaConv implementations across all tested configurations, with the largest gains in forward pass and gradient computation.
• Speedup is especially pronounced for large kernels, where the FFT method maintains constant performance regardless of kernel size due to fixed-size FFTs.
• The method is robust to varying input sizes and batch sizes, with consistent performance improvements across all tested settings.
• The algorithm’s performance is invariant to kernel size, enabling future exploration of larger kernels without computational penalty.