[Paper Review] Block Convolution: Towards Memory-Efficient Inference of Large-Scale CNNs on FPGA

Gang Li, Zejian Liu+2 more|arXiv (Cornell University)|May 19, 2021

Advanced Neural Network Applications17 references4 citations

TL;DR

This paper proposes block convolution, a hardware-friendly convolution operation that eliminates off-chip transfer of intermediate feature maps in FPGA-based CNN inference by splitting feature maps into independent spatial blocks. By removing tile boundary dependencies, it enables end-to-end layer fusion with minimal on-chip buffer usage, reducing off-chip memory transfers by over 99.9% and significantly improving energy efficiency and latency on memory-constrained FPGAs while maintaining comparable accuracy across ImageNet classification, COCO detection, and super-resolution tasks.

ABSTRACT

Deep convolutional neural networks have achieved remarkable progress in recent years. However, the large volume of intermediate results generated during inference poses a significant challenge to the accelerator design for resource-constraint FPGA. Due to the limited on-chip storage, partial results of intermediate layers are frequently transferred back and forth between on-chip memory and off-chip DRAM, leading to a non-negligible increase in latency and energy consumption. In this paper, we propose block convolution, a hardware-friendly, simple, yet efficient convolution operation that can completely avoid the off-chip transfer of intermediate feature maps at run-time. The fundamental idea of block convolution is to eliminate the dependency of feature map tiles in the spatial dimension when spatial tiling is used, which is realized by splitting a feature map into independent blocks so that convolution can be performed separately on individual blocks. We conduct extensive experiments to demonstrate the efficacy of the proposed block convolution on both the algorithm side and the hardware side. Specifically, we evaluate block convolution on 1) VGG-16, ResNet-18, ResNet-50, and MobileNet-V1 for ImageNet classification task; 2) SSD, FPN for COCO object detection task, and 3) VDSR for Set5 single image super-resolution task. Experimental results demonstrate that comparable or higher accuracy can be achieved with block convolution. We also showcase two CNN accelerators via algorithm/hardware co-design based on block convolution on memory-limited FPGAs, and evaluation shows that both accelerators substantially outperform the baseline without off-chip transfer of intermediate feature maps.

Motivation & Objective

• To address the high latency and energy consumption caused by frequent off-chip memory transfers during CNN inference on memory-constrained FPGAs.
• To eliminate data dependency between spatial tiles in conventional tiling-based CNN accelerators that forces repeated off-chip transfers of intermediate feature maps.
• To design a hardware-friendly convolution operation that enables end-to-end layer fusion without increasing on-chip buffer requirements.
• To demonstrate the broad applicability and efficiency of block convolution across diverse CNN architectures including VGG-16, ResNet, MobileNet, SSD, FPN, and VDSR.
• To validate the performance gains of block convolution through co-designed FPGA accelerators on Xilinx ZC706 and Ultra96 MPSoC platforms.

Proposed method

• Block convolution splits input feature maps into independent spatial blocks to decouple computation across tiles, removing boundary dependencies that require off-chip data transfer.
• The method enables multi-layer fusion by allowing each block to be processed end-to-end from input to output without storing intermediate results off-chip.
• The approach is compatible with existing hardware optimization techniques such as fixed-point quantization and pipelined processing on FPGA.
• A hardware accelerator is co-designed using high-level synthesis (HLS) to implement block convolution, with on-chip buffers for weights and intermediate results, and minimal off-chip data movement.
• The accelerator uses a single buffer for input and output tiles, eliminating the need for double buffering and scatter-gather memory access patterns.
• The design is evaluated on Xilinx Ultra96 MPSoC and ZC706 SoC platforms at 200MHz, with resource utilization and off-chip transfer volume measured.

Experimental results

Research questions

• RQ1 Can block convolution eliminate the need for off-chip transfer of intermediate feature maps in FPGA-based CNN inference?
• RQ2 How does block convolution affect model accuracy across diverse CNN architectures like VGG-16, ResNet, MobileNet, SSD, FPN, and VDSR?
• RQ3 To what extent can block convolution reduce on-chip buffer requirements and off-chip memory bandwidth pressure in resource-constrained FPGAs?
• RQ4 How does the performance of a block convolution-based accelerator compare to a baseline accelerator with conventional tiling in terms of latency, energy, and resource usage?
• RQ5 What is the impact of different blocking patterns and sizes on accuracy and hardware efficiency when using block convolution?

Key findings

• Block convolution reduces off-chip feature map transfer by over 99.9%, from 36.48 Gbits to 31.64 Mbits in the VDSR accelerator, drastically cutting energy consumption and bandwidth pressure.
• The BRAM usage is reduced from 352 out of 432 to 264 out of 432, demonstrating significant on-chip memory savings.
• The baseline accelerator with conventional tiling requires double buffering and complex memory access patterns, while the block convolution variant eliminates the need for such mechanisms.
• On the VDSR model, the block convolution accelerator achieves comparable PSNR performance with less than 1% degradation, confirming minimal accuracy loss.
• The method maintains high accuracy across diverse tasks: ImageNet classification (VGG-16, ResNet-18/50, MobileNet-V1), COCO object detection (SSD, FPN), and Set5 super-resolution (VDSR), with less than 1% mAP/PSNR degradation.
• The co-designed accelerators show substantial performance gains over the baseline, with reduced latency and energy consumption due to elimination of off-chip transfers and simplified memory access.