[Paper Review] Designing Efficient LLM Accelerators for Edge Devices

Jude Haris, Rappy Saha+2 more|arXiv (Cornell University)|Aug 1, 2024

Particle accelerators and beam dynamics4 citations

TL;DR

This paper proposes SECDA-LLM, a novel FPGA-based accelerator design platform that streamlines the development of efficient, hardware-software co-designed accelerators for large language model (LLM) inference on resource-constrained edge devices. By integrating the SECDA methodology with the llama.cpp inference framework, the platform enables rapid prototyping and deployment of custom accelerators, achieving an 11× speedup over dual-core ARM NEON CPU execution for the TinyLlama model using a block floating-point quantized MatMul accelerator.

ABSTRACT

The increase in open-source availability of Large Language Models (LLMs) has enabled users to deploy them on more and more resource-constrained edge devices to reduce reliance on network connections and provide more privacy. However, the high computation and memory demands of LLMs make their execution on resource-constrained edge devices challenging and inefficient. To address this issue, designing new and efficient edge accelerators for LLM inference is crucial. FPGA-based accelerators are ideal for LLM acceleration due to their reconfigurability, as they enable model-specific optimizations and higher performance per watt. However, creating and integrating FPGA-based accelerators for LLMs (particularly on edge devices) has proven challenging, mainly due to the limited hardware design flows for LLMs in existing FPGA platforms. To tackle this issue, in this paper we first propose a new design platform, named SECDA-LLM, that utilizes the SECDA methodology to streamline the process of designing, integrating, and deploying efficient FPGA-based LLM accelerators for the llama.cpp inference framework. We then demonstrate, through a case study, the potential benefits of SECDA-LLM by creating a new MatMul accelerator that supports block floating point quantized operations for LLMs. Our initial accelerator design, deployed on the PYNQ-Z1 board, reduces latency 1.7 seconds per token or ~2 seconds per word) by 11x over the dual-core Arm NEON-based CPU execution for the TinyLlama model.

Motivation & Objective

• To address the challenge of deploying large language models (LLMs) on resource-constrained edge devices due to high computational and memory demands.
• To reduce the complexity and development cost of creating custom FPGA-based accelerators for LLM inference by providing a streamlined design flow.
• To enable hardware-software co-design optimizations tailored to LLM inference workloads, particularly for quantized operations.
• To demonstrate the feasibility and performance gains of deploying a specialized FPGA accelerator for block floating-point quantized MatMul operations in LLMs.
• To provide a reusable, extensible platform for future development and collaboration in edge LLM acceleration.

Proposed method

• The SECDA-LLM platform integrates the SECDA methodology with the llama.cpp inference framework to enable hardware-software co-design for FPGA-based LLM accelerators.
• The platform supports AXI-API integration and profiling tools to streamline accelerator development, integration, and performance evaluation.
• A custom MatMul accelerator was designed with components including an instruction decoder, data mapper, scheduler, and Super-Block Vector Processor (SBVP) for efficient computation.
• The accelerator supports block floating-point quantization (Q3_K for weights, Q8_K for inputs), using super-blocks (SBs) with per-tile and global scaling factors to maintain model accuracy.
• The design maps super-blocks into on-chip buffers to minimize memory access latency and enable pipelined execution of dot products.
• The accelerator was deployed on the PYNQ-Z1 FPGA board and integrated with the llama.cpp runtime, offloading MatMul operations from the CPU.

Figure 1 : Overview of the SECDA methodology [ 11 ] . Components in the dashed lines correspond to simulation, and in the dotted lines to execution on real hardware.

Experimental results

Research questions

• RQ1 Can a unified design platform significantly reduce the complexity and time required to develop FPGA-based accelerators for LLM inference on edge devices?
• RQ2 To what extent can hardware-software co-design using SECDA-LLM improve inference performance for quantized LLMs on constrained edge hardware?
• RQ3 How effective is a custom FPGA accelerator for block floating-point quantized MatMul operations in reducing latency for LLM inference?
• RQ4 What performance gains can be achieved by offloading MatMul operations from a dual-core ARM NEON CPU to an FPGA accelerator on an edge device?
• RQ5 Can the SECDA-LLM platform support rapid prototyping and evaluation of new accelerator designs for emerging LLM quantization schemes?

Key findings

• The SECDA-LLM platform enables efficient, end-to-end development of FPGA-based LLM accelerators by integrating hardware design flows with the llama.cpp inference framework.
• The custom MatMul accelerator achieved an 11× speedup over dual-core ARM NEON CPU execution for the TinyLlama model, reducing latency to 1.7 seconds per token.
• The accelerator successfully supports block floating-point quantization (Q3_K and Q8_K), maintaining model accuracy while enabling efficient computation on limited-edge hardware.
• The use of super-blocks with per-tile and global scaling factors allowed precise computation with minimal precision loss, crucial for maintaining LLM performance.
• Execution profiling revealed that the accelerator design effectively minimized idle cycles and enabled high utilization of the FPGA resources during inference.
• The case study demonstrates that SECDA-LLM significantly reduces the barrier to entry for developing and deploying optimized LLM accelerators on edge FPGAs.

Figure 2 : Overview of SECDA-LLM. Key SECDA components are highlighted in orange, and the LLM components are highlighted in beige.