[Paper Review] Binary Complex Neural Network Acceleration on FPGA

Hongwu Peng, Shanglin Zhou+10 more|arXiv (Cornell University)|Aug 10, 2021

Neural Networks and Applications26 references4 citations

TL;DR

This paper proposes a hardware-software co-designed FPGA accelerator for Binary Complex Neural Networks (BCNN) that achieves over 5,800 frames per second inference throughput on the Alveo U280 platform. By combining structural pruning with a novel 2D convolution accelerator using high-level synthesis, the design reduces model size and memory bandwidth while maintaining high accuracy, outperforming a GPU by 1.58× on ResNet-18 and 1.51× on NIN-Net.

ABSTRACT

Being able to learn from complex data with phase information is imperative for many signal processing applications. Today' s real-valued deep neural networks (DNNs) have shown efficiency in latent information analysis but fall short when applied to the complex domain. Deep complex networks (DCN), in contrast, can learn from complex data, but have high computational costs; therefore, they cannot satisfy the instant decision-making requirements of many deployable systems dealing with short observations or short signal bursts. Recent, Binarized Complex Neural Network (BCNN), which integrates DCNs with binarized neural networks (BNN), shows great potential in classifying complex data in real-time. In this paper, we propose a structural pruning based accelerator of BCNN, which is able to provide more than 5000 frames/s inference throughput on edge devices. The high performance comes from both the algorithm and hardware sides. On the algorithm side, we conduct structural pruning to the original BCNN models and obtain 20 $ imes$ pruning rates with negligible accuracy loss; on the hardware side, we propose a novel 2D convolution operation accelerator for the binary complex neural network. Experimental results show that the proposed design works with over 90% utilization and is able to achieve the inference throughput of 5882 frames/s and 4938 frames/s for complex NIN-Net and ResNet-18 using CIFAR-10 dataset and Alveo U280 Board.

Motivation & Objective

• To enable real-time inference of complex-valued deep neural networks on resource-constrained edge devices.
• To reduce memory bandwidth and on-chip resource pressure in large DNNs by combining model compression and efficient hardware mapping.
• To design a high-throughput, low-latency FPGA accelerator for BCNNs using structural pruning and custom convolutional kernel architecture.
• To evaluate the performance and efficiency of the proposed BCNN accelerator on real-world datasets like CIFAR-10 and ImageNet.

Proposed method

• Structural pruning is applied to BCNN models using Surrogate Lagrangian Relaxation (SLR), achieving up to 20× pruning rates with minimal accuracy loss.
• Binarized weights are quantized using the Straight-Through Estimator (STE), enabling efficient binary operations in hardware.
• A novel 2D convolution accelerator is designed to exploit parallelism and minimize pipeline initiation interval for binary complex operations.
• High-Level Synthesis (HLS) is used to implement the accelerator on Xilinx Alveo U280 FPGA, with resource scheduling to maximize throughput.
• The design targets on-chip memory usage to avoid off-chip memory bottlenecks, leveraging high on-chip bandwidth and low latency.
• Hardware evaluation is conducted using CIFAR-10 and ImageNet datasets, with throughput and latency measured on Alveo U280 and RTX 6000 GPU for comparison.

Experimental results

Research questions

• RQ1 Can structural pruning combined with binary quantization achieve high compression rates in complex-valued neural networks with minimal accuracy degradation?
• RQ2 How can an FPGA-based accelerator be architected to maximize throughput and hardware utilization for binary complex convolution operations?
• RQ3 What is the performance gain of an FPGA-accelerated BCNN over a GPU-based implementation in terms of inference throughput and energy efficiency?
• RQ4 How does the choice of pooling layer (spectral, average, max) affect the accuracy and efficiency of BCNN models?
• RQ5 To what extent can on-chip memory utilization be maximized to avoid off-chip memory access bottlenecks in large BCNN models?

Key findings

• The proposed BCNN accelerator achieves 5,882 frames per second throughput for NIN-Net and 4,938 frames per second for ResNet-18 on the Alveo U280 FPGA.
• The design achieves a 20× model compression rate with only 1.9% accuracy drop on CIFAR-10 for complex NIN-Net and 2.8% for complex ResNet-18.
• The FPGA implementation achieves 1.51× and 1.58× speedup over a single RTX 6000 GPU for NIN-Net and ResNet-18, respectively.
• The hardware utilization reaches over 90%, with LUTs as the primary resource bottleneck, enabling 9 and 8 parallel kernels for NIN-Net and ResNet-18, respectively.
• Average pooling outperforms spectral and max pooling in accuracy with acceptable complexity, and is therefore selected for the final model.
• On ImageNet, the complex ResNetE-18 model maintains 83.46% top-5 accuracy in the original form, with 78.38% and 71.69% after pruning and pruning+quantization, respectively.