[Paper Review] FFT-Based Deep Learning Deployment in Embedded Systems

Sheng Lin, Ning Liu+5 more|arXiv (Cornell University)|Dec 13, 2017

Advanced Neural Network Applications25 references4 citations

TL;DR

This paper proposes an FFT-based deep neural network (DNN) framework for embedded systems that reduces computational and storage complexity through Fast Fourier Transform-based weight matrices, achieving high inference speed with minimal accuracy loss. The method enables efficient deployment on ARM-based mobile platforms, demonstrating up to 130% faster inference in C++ compared to Java, and outperforms IBM TrueNorth in speed on MNIST despite using far fewer cores.

ABSTRACT

Deep learning has delivered its powerfulness in many application domains, especially in image and speech recognition. As the backbone of deep learning, deep neural networks (DNNs) consist of multiple layers of various types with hundreds to thousands of neurons. Embedded platforms are now becoming essential for deep learning deployment due to their portability, versatility, and energy efficiency. The large model size of DNNs, while providing excellent accuracy, also burdens the embedded platforms with intensive computation and storage. Researchers have investigated on reducing DNN model size with negligible accuracy loss. This work proposes a Fast Fourier Transform (FFT)-based DNN training and inference model suitable for embedded platforms with reduced asymptotic complexity of both computation and storage, making our approach distinguished from existing approaches. We develop the training and inference algorithms based on FFT as the computing kernel and deploy the FFT-based inference model on embedded platforms achieving extraordinary processing speed.

Motivation & Objective

• To address the challenge of deploying large, accurate DNN models on resource-constrained embedded systems with limited memory and computation.
• To reduce both computational complexity and model storage requirements in DNNs without significant accuracy degradation.
• To develop a novel FFT-based DNN framework that enables efficient training and inference on mobile and embedded platforms.
• To demonstrate superior inference performance on ARM-based embedded systems compared to prior ASIC-based solutions like IBM TrueNorth.

Proposed method

• The method employs Fast Fourier Transform (FFT) as the core computing kernel for both training and inference, replacing standard dense matrix operations.
• Weight matrices in fully connected and convolutional layers are structured using circulant or block-circulant matrices to exploit FFT's efficiency.
• The framework uses FFT-based matrix multiplication to reduce the asymptotic complexity of both computation and storage from O(n²) to O(n log n).
• The approach enables model compression by representing large weight matrices with fewer parameters via circulant structure, reducing parameter count by up to a factor of n.
• The model is trained end-to-end with backpropagation adapted to the FFT-based computation, preserving accuracy through fine-tuning.
• Inference is deployed on ARM-based mobile platforms using C++ and Java implementations, with OpenCV for image processing.

Experimental results

Research questions

• RQ1 Can FFT-based weight matrix representation significantly reduce the computational and storage complexity of DNNs on embedded systems?
• RQ2 How does the FFT-based DNN framework compare in inference speed and accuracy to existing embedded and ASIC-based solutions like IBM TrueNorth?
• RQ3 To what extent can model compression via circulant matrices preserve classification accuracy while reducing parameter count?
• RQ4 What performance differences arise between C++ and Java implementations of the FFT-based inference on mobile platforms?

Key findings

• The FFT-based DNN framework reduces model storage requirements and computational complexity by leveraging circulant matrix structures and FFT operations.
• On the MNIST dataset, the framework achieves 10× faster inference than IBM TrueNorth on a mobile platform with only a minor accuracy reduction.
• On CIFAR-10, the C++ implementation achieves 8,912 μs per image inference time, which is 130% faster than the Java implementation (21,032 μs).
• The C++ implementation on the Honor 6X device achieves 8,244 μs per image, significantly outperforming the Java version (19,785 μs).
• Despite using 500–1,000 times fewer cores than IBM TrueNorth, the proposed framework achieves competitive performance on CIFAR-10 with 80.2% accuracy.
• The performance gap between C++ and Java implementations is attributed to memory management constraints and data type conversion overhead in Android.