[Paper Review] Automated flow for compressing convolution neural networks for efficient edge-computation with FPGA

Farhan Shafiq, Takato Yamada+2 more|arXiv (Cornell University)|Dec 18, 2017

Advanced Neural Network Applications7 references3 citations

TL;DR

This paper presents an automated, end-to-end flow for compressing and deploying binarized convolutional neural networks (CNNs) on FPGA-based edge devices. It quantizes weights to 1-bit and activations to 2-bit, generates optimized C code and FPGA accelerators via high-level synthesis, and achieves 11.5× speedup on binarized convolution over mobile CPU, with full synthesis in under one hour—demonstrated on YOLOv2 deployed on a Cyclone-V FPGA SoC.

ABSTRACT

Deep convolutional neural networks (CNN) based solutions are the current state- of-the-art for computer vision tasks. Due to the large size of these models, they are typically run on clusters of CPUs or GPUs. However, power requirements and cost budgets can be a major hindrance in adoption of CNN for IoT applications. Recent research highlights that CNN contain significant redundancy in their structure and can be quantized to lower bit-width parameters and activations, while maintaining acceptable accuracy. Low bit-width and especially single bit-width (binary) CNN are particularly suitable for mobile applications based on FPGA implementation, due to the bitwise logic operations involved in binarized CNN. Moreover, the transition to lower bit-widths opens new avenues for performance optimizations and model improvement. In this paper, we present an automatic flow from trained TensorFlow models to FPGA system on chip implementation of binarized CNN. This flow involves quantization of model parameters and activations, generation of network and model in embedded-C, followed by automatic generation of the FPGA accelerator for binary convolutions. The automated flow is demonstrated through implementation of binarized "YOLOV2" on the low cost, low power Cyclone- V FPGA device. Experiments on object detection using binarized YOLOV2 demonstrate significant performance benefit in terms of model size and inference speed on FPGA as compared to CPU and mobile CPU platforms. Furthermore, the entire automated flow from trained models to FPGA synthesis can be completed within one hour.

Motivation & Objective

• Address the challenge of deploying large, power-hungry deep CNNs on energy- and cost-constrained edge devices.
• Reduce model size and inference latency for CNNs on low-power FPGAs through automated quantization and hardware generation.
• Enable efficient FPGA acceleration of binarized CNNs by optimizing data ordering and memory access patterns.
• Develop a fully automated pipeline from trained TensorFlow models to FPGA-synthesizable hardware accelerators for edge inference.

Proposed method

• Quantize trained TensorFlow models to 1-bit weights and 2-bit activations, excluding first and last layers.
• Parse the quantized model from TensorFlow protocol buffer format and apply graph-level transformations to replace floating-point operations with bit-level equivalents.
• Generate compact, self-contained embedded-C code for the quantized network, enabling efficient on-chip execution.
• Automatically generate an FPGA accelerator using high-level synthesis (HLS), tailored to the model’s computational and memory requirements.
• Implement a depth-first data-ordering strategy to improve memory access continuity and reduce bit-masking overhead in local RAM.
• Enable inter-kernel parallelism to increase throughput and improve burst memory performance during convolution operations.

Experimental results

Research questions

• RQ1 Can an automated flow significantly reduce model size and inference latency for binarized CNNs on low-power FPGAs?
• RQ2 How effective is depth-first data ordering in improving memory access efficiency and reducing hardware complexity in FPGA-based binarized CNN accelerators?
• RQ3 To what extent can FPGA-based acceleration outperform mobile and general-purpose CPUs in terms of speed and energy efficiency for binarized CNN inference?
• RQ4 What is the end-to-end time cost of transforming a trained TensorFlow model into a synthesized FPGA accelerator with minimal human intervention?

Key findings

• The proposed automated flow converts a trained TensorFlow model into a fully synthesized FPGA accelerator in approximately one hour.
• The compressed binarized YOLOv2 model is reduced to 8.26 MB (32× smaller than the original 255.82 MB model).
• The FPGA-accelerated binarized convolution achieves 11.5× speedup over a mobile CPU and 1.21× speedup over a high-end desktop CPU (Core i7-6800K).
• The total inference time on the FPGA-SoC is 5.34× faster than on a mobile CPU, though still 1.78× slower than the Core i7 CPU.
• The depth-first data-ordering strategy enables more efficient bit-packing and reduces the need for multiple memory accesses and masking operations.
• Inter-kernel parallelism combined with optimized memory access results in better burst performance and cleaner memory write circuitry on the FPGA.