[Paper Review] Accelerating Framework of Transformer by Hardware Design and Model Compression Co-Optimization

Panjie Qi, Edwin H.‐M. Sha+6 more|arXiv (Cornell University)|Oct 19, 2021

Advanced Data Compression Techniques4 citations

TL;DR

This paper proposes a co-optimized hardware-software framework that jointly accelerates Transformer inference by co-designing model compression via hierarchical pruning (HP) and FPGA-specific hardware acceleration. The framework automatically selects the optimal device based on latency and accuracy constraints, achieving up to 37× speedup over CPU and 1.9× over GPU, with high sparsity (up to 98%) and efficient memory usage through custom sparse matrix storage.

ABSTRACT

State-of-the-art Transformer-based models, with gigantic parameters, are difficult to be accommodated on resource constrained embedded devices. Moreover, with the development of technology, more and more embedded devices are available to run a Transformer model. For a Transformer model with different constraints (tight or loose), it can be deployed onto devices with different computing power. However, in previous work, designers did not choose the best device among multiple devices. Instead, they just used an existing device to deploy model, which was not necessarily the best fit and may lead to underutilization of resources. To address the deployment challenge of Transformer and the problem to select the best device, we propose an algorithm & hardware closed-loop acceleration framework. Given a dataset, a model, latency constraint LC and accuracy constraint AC, our framework can provide a best device satisfying both constraints. In order to generate a compressed model with high sparsity ratio, we propose a novel pruning technique, hierarchical pruning (HP). We optimize the sparse matrix storage format for HP matrix to further reduce memory usage for FPGA implementation. We design a accelerator that takes advantage of HP to solve the problem of concurrent random access. Experiments on Transformer and TinyBert model show that our framework can find different devices for various LC and AC, covering from low-end devices to high-end devices. Our HP can achieve higher sparsity ratio and is more flexible than other sparsity pattern. Our framework can achieve 37x, 1.9x, 1.7x speedup compared to CPU, GPU and FPGA, respectively.

Motivation & Objective

• To address the challenge of deploying large-scale Transformer models on resource-constrained embedded devices with varying performance requirements.
• To overcome the inefficiency of prior sequential design flows that do not co-optimize model compression and hardware selection.
• To enable automatic device selection from a diverse set of hardware platforms (e.g., FPGAs, mobile devices) based on user-specified latency and accuracy constraints.
• To develop a high-sparsity, flexible pruning method (hierarchical pruning) that supports efficient sparse matrix storage and hardware mapping.
• To design a hardware accelerator that exploits the sparsity pattern of HP to reduce memory access overhead and improve throughput.

Proposed method

• Proposes an algorithm ⇄ hardware closed-loop framework that co-optimizes model sparsity and hardware deployment under user-defined latency (LC) and accuracy (AC) constraints.
• Introduces hierarchical pruning (HP), a novel structured pruning technique that enables high sparsity (up to 98%) with flexible, block-level sparsity patterns across different layers.
• Designs a custom sparse matrix storage format optimized for HP sparsity patterns to minimize memory footprint on FPGAs.
• Develops a dedicated FPGA accelerator that supports concurrent random access to sparse weights, improving memory bandwidth utilization.
• Employs reinforcement learning (RL) to explore the search space of sparsity ratios and device configurations, guided by LC and AC constraints.
• Uses FLOPS and resource utilization as metrics to evaluate and compare performance across CPU, GPU, and FPGA platforms.

Experimental results

Research questions

• RQ1 What is the optimal hardware platform for deploying a Transformer model under specific latency and accuracy constraints?
• RQ2 How can model compression be co-designed with hardware acceleration to maximize inference speed and resource utilization?
• RQ3 Can a hierarchical pruning strategy achieve higher sparsity ratios while preserving model accuracy compared to existing structured pruning methods?
• RQ4 How does the proposed sparse storage format reduce memory usage and improve performance on FPGAs?
• RQ5 Can the framework automatically select the best device from a heterogeneous set of platforms (e.g., ZCU102, Alveo U200) for different constraint sets?

Key findings

• The framework successfully identifies the best device for various LC and AC constraints, ranging from low-end FPGAs (ZCU102) to high-end devices (Alveo U200), enabling deployment across diverse embedded platforms.
• Hierarchical pruning achieves a sparsity ratio of up to 98%, significantly higher than the 90% limit of existing methods like VW, with better accuracy retention at high sparsity levels.
• The FPGA accelerator achieves 37× speedup over CPU, 1.9× over GPU, and 1.7× over a prior FPGA implementation (FTRANS), demonstrating superior efficiency.
• For the same constraint (e.g., 50ms, 80% accuracy), the framework maps to lower-end FPGAs (ZCU102) through model compression, enabling reuse across different application scenarios.
• The RL-guided search concentrates solutions near the target constraint (26ms, 96%), and resource utilization is used as a tiebreaker to select the most cost-effective device.
• When sparsity is 88%, a 40% backbone model achieves 96.4% accuracy, outperforming models with 60% and 70% backbone sparsity, indicating that lower initial sparsity preserves accuracy better under high compression.