[Paper Review] Hardware Acceleration of LLMs: A comprehensive survey and comparison

Nikoletta Koilia, Christoforos Kachris|arXiv (Cornell University)|Sep 5, 2024

Parallel Computing and Optimization Techniques4 citations

TL;DR

This paper presents a comprehensive survey and fair comparison of hardware accelerators for Large Language Models (LLMs), using extrapolation to standardize performance and energy efficiency across different process technologies (especially 16nm). It evaluates FPGAs, ASICs, GPUs, and in-memory architectures, demonstrating that ASICs deliver the highest performance (up to 363,647 GOPS), while in-memory accelerators offer superior energy efficiency, enabling more accurate cross-platform benchmarking despite technological disparities.

ABSTRACT

Large Language Models (LLMs) have emerged as powerful tools for natural language processing tasks, revolutionizing the field with their ability to understand and generate human-like text. In this paper, we present a comprehensive survey of the several research efforts that have been presented for the acceleration of transformer networks for Large Language Models using hardware accelerators. The survey presents the frameworks that have been proposed and then performs a qualitative and quantitative comparison regarding the technology, the processing platform (FPGA, ASIC, In-Memory, GPU), the speedup, the energy efficiency, the performance (GOPs), and the energy efficiency (GOPs/W) of each framework. The main challenge in comparison is that every proposed scheme is implemented on a different process technology making hard a fair comparison. The main contribution of this paper is that we extrapolate the results of the performance and the energy efficiency on the same technology to make a fair comparison; one theoretical and one more practical. We implement part of the LLMs on several FPGA chips to extrapolate the results to the same process technology and then we make a fair comparison of the performance.

Motivation & Objective

• To address the challenge of unfair performance comparisons across LLM hardware accelerators due to varying process technologies.
• To enable a standardized benchmarking framework by extrapolating performance and energy efficiency to a common 16nm process technology.
• To evaluate and compare the performance and energy efficiency of accelerators across FPGA, ASIC, GPU, and in-memory platforms.
• To provide researchers with a reliable, technology-agnostic basis for selecting or designing LLM accelerators.

Proposed method

• The authors use scaling equations from prior work (e.g., Stillmaker & Baas, 2017) to extrapolate performance and energy efficiency to a 16nm process technology.
• They perform theoretical extrapolation on FPGA and ASIC accelerators using published performance data and process node characteristics.
• An experimental validation is conducted using a VHDL-based matrix multiplication IP core on FPGAs of varying process nodes (180nm to 20nm) to measure maximum clock frequency.
• The experimental results are used to validate and refine the theoretical extrapolation model for FPGA-based accelerators.
• Performance is quantified in GOPs (giga operations per second), and energy efficiency in GOPs/W, enabling cross-platform comparison.
• The comparison includes accelerators targeting LLMs such as Swin-T, Swin-B, Swin-S, BETA, Me-ViT, FTRANS, Via, STA-4, STA-8, and AccelTran (edge/server) ASICs.

Figure 1: Performance per process technology

Experimental results

Research questions

• RQ1 How can performance and energy efficiency of LLM accelerators be fairly compared across different process technologies?
• RQ2 What is the relative performance and energy efficiency of FPGA-, ASIC-, GPU-, and in-memory-based LLM accelerators when standardized to a 16nm process node?
• RQ3 To what extent do theoretical scaling models accurately predict FPGA performance across different process nodes?
• RQ4 Which hardware platform delivers the highest absolute performance (GOPs) and energy efficiency (GOPs/W) for LLM inference after standardization?
• RQ5 How does experimental measurement of clock frequency on FPGAs of varying process nodes support theoretical extrapolation?

Key findings

• After extrapolation to 16nm, the AccelTran (server) ASIC achieves the highest performance at 363,647 GOPs, significantly outperforming other platforms.
• The Me-ViT FPGA accelerator achieves 2,682 GOPs on 16nm, demonstrating high performance for FPGA-based designs.
• The BETA FPGA accelerator achieves 1,436 GOPs on 16nm, indicating strong performance for attention-heavy LLM workloads.
• In-memory accelerators show superior energy efficiency compared to other platforms, though absolute performance is not the focus of this study.
• The experimental FPGA measurements (180nm to 20nm) confirm the validity of the theoretical scaling model, with clock frequency decreasing predictably as process technology scales up.
• The study establishes that fair comparison across accelerators is only possible through technology normalization, as raw performance data is misleading when derived from different process nodes.

Figure 2: Performance (Log) per process technology