[Paper Review] Low Latency CMOS Hardware Acceleration for Fully Connected Layers in Deep Neural Networks

Nick Iliev, Amit Ranjan Trivedi|arXiv (Cornell University)|Nov 25, 2020

Advanced Neural Network Applications4 citations

TL;DR

This paper presents FC-ACCL, a low-latency CMOS ASIC accelerator for fully connected (FC) layers in deep neural networks, using 128 parallel 8x8 or 16x16 processing elements (PEs) and 128 High Bandwidth Memory (HBM) units to store and stream weights in a novel column-row-column schedule. The design achieves 48.4 GOPS at 100 MHz for a 4096×1000 FC8 layer, outperforming a prior 28.8 GOPS accelerator at 150 MHz by maximizing parallelism and minimizing memory access latency.

ABSTRACT

We present a novel low latency CMOS hardware accelerator for fully connected (FC) layers in deep neural networks (DNNs). The FC accelerator, FC-ACCL, is based on 128 8x8 or 16x16 processing elements (PEs) for matrix-vector multiplication, and 128 multiply-accumulate (MAC) units integrated with 128 High Bandwidth Memory (HBM) units for storing the pretrained weights. Micro-architectural details for CMOS ASIC implementations are presented and simulated performance is compared to recent hardware accelerators for DNNs for AlexNet and VGG 16. When comparing simulated processing latency for a 4096-1000 FC8 layer, our FC-ACCL is able to achieve 48.4 GOPS (with a 100 MHz clock) which improves on a recent FC8 layer accelerator quoted at 28.8 GOPS with a 150 MHz clock. We have achieved this considerable improvement by fully utilizing the HBM units for storing and reading out column-specific FClayer weights in 1 cycle with a novel colum-row-column schedule, and implementing a maximally parallel datapath for processing these weights with the corresponding MAC and PE units. When up-scaled to 128 16x16 PEs, for 16x16 tiles of weights, the design can reduce latency for the large FC6 layer by 60 % in AlexNet and by 3 % in VGG16 when compared to an alternative EIE solution which uses compression.

Motivation & Objective

• To address the high latency of fully connected (FC) layers in DNN inference, especially in models like AlexNet and VGG16.
• To reduce memory access bottlenecks in FC layers by enabling single-cycle column-specific weight access.
• To design a highly parallel, area-efficient CMOS ASIC architecture that maximizes throughput and minimizes latency for FC layers.
• To outperform existing hardware accelerators in processing latency while maintaining high utilization of on-chip resources.

Proposed method

• The FC-ACCL architecture employs 128 parallel 8x8 or 16x16 processing elements (PEs) for matrix-vector multiplication in fully connected layers.
• It integrates 128 High Bandwidth Memory (HBM) units to store and stream the full weight matrix, enabling column-specific access in a single cycle.
• A novel column-row-column memory access schedule is used to optimize data reuse and reduce idle cycles in the MAC and PE units.
• The datapath is fully parallelized, with each PE and MAC unit operating in lockstep to maximize throughput.
• The design uses a tiled architecture where weights are processed in 16x16 blocks, enabling efficient scaling to larger FC layers.
• The micro-architecture is optimized for CMOS ASIC implementation, with detailed timing and area analysis for 128 PEs.

Experimental results

Research questions

• RQ1 Can a CMOS ASIC accelerator achieve lower latency than existing FC layer accelerators by optimizing memory access patterns?
• RQ2 How does a column-row-column memory scheduling strategy impact the latency and throughput of FC layer inference?
• RQ3 To what extent can parallelism in PEs and MAC units be maximized to improve GOPS performance?
• RQ4 How does the proposed architecture scale in latency reduction for large FC layers like FC6 in AlexNet and FC8 in VGG16?
• RQ5 What is the performance gain of the FC-ACCL design compared to a compressed EIE-based solution in terms of latency and throughput?

Key findings

• The FC-ACCL accelerator achieves 48.4 GOPS at a 100 MHz clock for a 4096×1000 FC8 layer, significantly outperforming a prior accelerator that achieved 28.8 GOPS at 150 MHz.
• The novel column-row-column memory access schedule enables single-cycle column-specific weight access, reducing memory latency and improving utilization.
• Scaling to 128 16x16 PEs reduces latency by 60% for the large FC6 layer in AlexNet compared to an alternative EIE-based solution.
• For the FC8 layer in VGG16, the design reduces latency by 3% compared to the same EIE-based solution.
• The architecture achieves high throughput by fully utilizing the 128 MAC units and 128 PEs in a maximally parallel datapath, minimizing idle cycles.
• The design demonstrates that memory bandwidth and scheduling are critical bottlenecks in FC layer acceleration, and their optimization yields substantial performance gains.