[Paper Review] MSP: An FPGA-Specific Mixed-Scheme, Multi-Precision Deep Neural Network Quantization Framework

Sung-En Chang, Yanyu Li+5 more|arXiv (Cornell University)|Sep 16, 2020

Advanced Neural Network Applications47 references4 citations

TL;DR

This paper proposes MSP, a novel FPGA-specific mixed-scheme, multi-precision quantization framework that combines sum-of-power-of-two (SPoT) and fixed-point quantization to jointly exploit LUTs and DSPs on FPGAs. By enabling intra-layer multi-precision quantization with 5% 8-bit weights, MSP achieves 70.47% Top1 accuracy on ImageNet with 3.53× speedup over fixed-point quantization, preserving accuracy while maximizing hardware utilization.

ABSTRACT

With the tremendous success of deep learning, there exists imminent need to deploy deep learning models onto edge devices. To tackle the limited computing and storage resources in edge devices, model compression techniques have been widely used to trim deep neural network (DNN) models for on-device inference execution. This paper targets the commonly used FPGA (field programmable gate array) devices as the hardware platforms for DNN edge computing. We focus on the DNN quantization as the main model compression technique, since DNN quantization has been of great importance for the implementations of DNN models on the hardware platforms. The novelty of this work comes in twofold: (i) We propose a mixed-scheme DNN quantization method that incorporates both the linear and non-linear number systems for quantization, with the aim to boost the utilization of the heterogeneous computing resources, i.e., LUTs (look up tables) and DSPs (digital signal processors) on an FPGA. Note that all the existing (single-scheme) quantization methods can only utilize one type of resources (either LUTs or DSPs for the MAC (multiply-accumulate) operations in deep learning computations. (ii) We use a quantization method that supports multiple precisions along the intra-layer dimension, while the existing quantization methods apply multi-precision quantization along the inter-layer dimension. The intra-layer multi-precision method can uniform the hardware configurations for different layers to reduce computation overhead and at the same time preserve the model accuracy as the inter-layer approach.

Motivation & Objective

• To address the challenge of deploying low-precision DNNs on resource-constrained FPGAs with high hardware efficiency and minimal accuracy loss.
• To overcome the limitation of single-scheme quantization (e.g., only LUTs or only DSPs) by co-optimizing heterogeneous FPGA resources (LUTs and DSPs).
• To reduce hardware configuration complexity and computation overhead by enabling intra-layer multi-precision quantization instead of inter-layer approaches.
• To achieve lossless accuracy performance even when quantizing first and last layers, which are typically sensitive to precision loss.
• To provide a unified, hardware-friendly quantization framework that supports mixed-precision and mixed-scheme operations for efficient FPGA deployment.

Proposed method

• Proposes a sum-of-power-of-two (SPoT) quantization scheme that enables bit-shift and add operations using LUTs, reducing reliance on DSPs and minimizing accuracy degradation compared to vanilla power-of-two schemes.
• Introduces a mixed-scheme (MS) quantization that dynamically combines SPoT and fixed-point quantization based on FPGA resource availability, maximizing utilization of both LUTs and DSPs.
• Develops an intra-layer multi-precision (MP) strategy that assigns 5% of weights in each layer to 8-bit precision, preserving model accuracy without requiring different bit-widths across layers.
• Employs a hybrid quantization ratio (e.g., 65% SPoT, 30% fixed-point, 5% 8-bit) optimized via FPGA resource utilization analysis to balance accuracy and performance.
• Uses a unified framework to integrate SPoT, fixed-point, and 8-bit quantization within the same layer, enabling lossless accuracy and efficient hardware mapping.
• Validates the framework on Xilinx XC7Z020 and XC7Z045 FPGAs with 100MHz clock, measuring throughput, latency, and resource utilization across ResNet-18 and MobileNet-v2.

Experimental results

Research questions

• RQ1 Can a mixed-scheme quantization approach that combines SPoT and fixed-point quantization improve FPGA resource utilization compared to single-scheme methods?
• RQ2 Does intra-layer multi-precision quantization outperform inter-layer multi-precision in terms of hardware efficiency and accuracy preservation?
• RQ3 Can SPoT quantization reduce accuracy degradation while maintaining computational efficiency through bit-shift operations on LUTs?
• RQ4 To what extent can the proposed MSP framework achieve speedup and maintain accuracy on real FPGA platforms like XC7Z020 and XC7Z045?
• RQ5 How does the combination of mixed-scheme and intra-layer multi-precision quantization affect throughput and resource utilization in DNN inference on FPGAs?

Key findings

• MSP achieves 70.47% Top1 accuracy on ImageNet using ResNet-18, outperforming baseline floating-point models by 0.71% in accuracy.
• The framework achieves a 3.53× speedup in end-to-end inference time on FPGA compared to fixed-point quantization, with throughput increasing from 120.5 GOPS to 325.0 GOPS on XC7Z045 for ResNet-18.
• DSP utilization remains at 100% across all configurations, while LUT utilization increases from 19% to 69% when SPoT ratio is increased, indicating effective resource balancing.
• The intra-layer multi-precision strategy enables lossless accuracy even when first and last layers are quantized, unlike prior methods that avoid quantizing these layers.
• Ablation studies confirm that mixed-scheme quantization (MS) improves speed and accuracy over single-scheme methods, and intra-layer MP further enhances performance without sacrificing accuracy.
• The optimal quantization ratio (65% SPoT, 30% fixed-point, 5% 8-bit) was validated via FPGA resource utilization and performance profiling, achieving the best trade-off between efficiency and accuracy.