[Paper Review] Accelerating CNN inference on FPGAs: A Survey

Kamel Abdelouahab, Maxime Pelcat+2 more|arXiv (Cornell University)|May 26, 2018

Advanced Neural Network Applications47 references102 citations

TL;DR

A comprehensive survey of FPGA-based CNN inference accelerators, detailing algorithmic transforms (GEMM, Winograd, FFT), data-path optimizations, approximate computing, and design-space exploration strategies to map CNNs efficiently onto FPGA hardware.

ABSTRACT

Convolutional Neural Networks (CNNs) are currently adopted to solve an ever greater number of problems, ranging from speech recognition to image classification and segmentation. The large amount of processing required by CNNs calls for dedicated and tailored hardware support methods. Moreover, CNN workloads have a streaming nature, well suited to reconfigurable hardware architectures such as FPGAs. The amount and diversity of research on the subject of CNN FPGA acceleration within the last 3 years demonstrates the tremendous industrial and academic interest. This paper presents a state-of-the-art of CNN inference accelerators over FPGAs. The computational workloads, their parallelism and the involved memory accesses are analyzed. At the level of neurons, optimizations of the convolutional and fully connected layers are explained and the performances of the different methods compared. At the network level, approximate computing and datapath optimization methods are covered and state-of-the-art approaches compared. The methods and tools investigated in this survey represent the recent trends in FPGA CNN inference accelerators and will fuel the future advances on efficient hardware deep learning.

Motivation & Objective

• Summarize the state-of-the-art in FPGA-based CNN inference accelerators and identify common optimization strategies.
• Analyze how convolutional and fully-connected layers are accelerated on FPGAs through algorithmic transforms and data-path design.
• Discuss memory access patterns and caching strategies that impact performance and energy efficiency.
• Compare approaches to approximate computing and model/data optimizations in the FPGA context.
• Provide guidance on design-space trade-offs (unrolling, tiling, and hardware generation) for FPGA CNN implementations.

Proposed method

• Review CNN inference workloads and parallelism patterns to understand hardware mapping challenges.
• Classify and explain algorithmic optimizations used to accelerate conv and FC layers on FPGAs (GEMM, Winograd, FFT).
• Describe data-path optimization techniques (systolic arrays, SIMD accelerators, loop unrolling/tiling, caching).
• Discuss design-space exploration methods (Roofline-guided brute force and heuristics) for configuring FPGA accelerators.
• Survey data-path and hardware generation approaches (HLS/OpenCL, RTL, DSL) used to implement FPGA CNN accelerators.

Experimental results

Research questions

• RQ1 What are the dominant algorithmic transforms used to map CNN layers onto FPGAs and how do they impact performance and memory use?
• RQ2 How do data-path optimizations and loop transformations enable efficient CNN inference on resource-limited FPGA devices?
• RQ3 What role do memory bandwidth and on-chip buffers play in FPGA CNN throughput and energy efficiency?
• RQ4 How do design-space exploration methods guide the configuration of FPGA accelerators for CNN workloads?
• RQ5 What trends in hardware generation (HLS/OpenCL/RTL) support scalable FPGA CNN implementations?

Key findings

• GEMM-based, Winograd, and FFT transformations are central techniques for mapping CNN layers to FPGA accelerators, each with trade-offs in throughput and memory access.
• Data-path optimizations, including loop unrolling and tiling, are critical to balance compute and memory and to fit on-chip buffers.
• Memory bandwidth and cache hierarchy are major bottlenecks; effective on-chip buffers significantly reduce external memory traffic and energy.
• Design-space exploration, often Roofline-guided, is essential to select unrolling/tiling parameters that maximize throughput within FPGA resource limits.
• Various FPGA implementations (HLS/OpenCL and RTL) demonstrate substantial performance gains by reusing filters across batches and by fusing layers.