[論文レビュー] FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
tldr: FlexGenは、GPU/CPU/diskメモリを統合し、テンソル配置を線形計画法の探索で最適化し、4-bit 重み/KVキャッシュ圧縮を適用することで、単一の一般用途GPU上で大規模言語モデルの高スループット生成推論を可能にするオフロードベースのフレームワークを提示する。
The high computational and memory requirements of large language model (LLM) inference make it feasible only with multiple high-end accelerators. Motivated by the emerging demand for latency-insensitive tasks with batched processing, this paper initiates the study of high-throughput LLM inference using limited resources, such as a single commodity GPU. We present FlexGen, a high-throughput generation engine for running LLMs with limited GPU memory. FlexGen can be flexibly configured under various hardware resource constraints by aggregating memory and computation from the GPU, CPU, and disk. By solving a linear programming problem, it searches for efficient patterns to store and access tensors. FlexGen further compresses the weights and the attention cache to 4 bits with negligible accuracy loss. These techniques enable FlexGen to have a larger space of batch size choices and thus significantly increase maximum throughput. As a result, when running OPT-175B on a single 16GB GPU, FlexGen achieves significantly higher throughput compared to state-of-the-art offloading systems, reaching a generation throughput of 1 token/s for the first time with an effective batch size of 144. On the HELM benchmark, FlexGen can benchmark a 30B model with a 16GB GPU on 7 representative sub-scenarios in 21 hours. The code is available at https://github.com/FMInference/FlexGen
研究の動機と目的
- Motivate throughput-oriented generative inference under limited GPU memory for LLMs.
- Develop a formal offloading strategy that optimizes tensor placement and computation schedule across GPU, CPU, and disk.
- Propose a 4-bit group-wise quantization approach to compress weights and KV cache without retraining.
- Show that coordinated offloading and compression yield substantial throughput gains on OPT models, especially at very large scales.
提案手法
- Formally define a search space over compute schedule (row-by-row vs column-by-column vs zig-zag block), tensor placement (weights/activations/KV cache across GPU/CPU/disk), and computation delegation (CPU vs GPU for certain tasks).
- Use a linear-programming-based policy search to optimize throughput under hardware constraints with an analytical cost model estimating I/O and compute costs.
- Propose a 4-bit fine-grained group-wise quantization for both weights and KV cache to reduce I/O and memory, with dequantization to FP16 for computation.
- Enable approximate methods (group-wise quantization and sparse attention) to further boost throughput with negligible accuracy loss.
- Extend the approach to multi-GPU setups using pipeline parallelism to achieve potential super-linear scaling in decoding throughput.
実験結果
リサーチクエスチョン
- RQ1How can we achieve high-throughput LLM inference on a single commodity GPU by combining GPU/CPU/disk memory?
- RQ2What offloading strategies and tensor placement policies maximize throughput for generative inference with large models like OPT-175B?
- RQ3Can 4-bit quantization of weights and KV cache be realized without retraining to substantially reduce memory and I/O costs?
- RQ4What is the throughput advantage of FlexGen compared to existing offloading systems on large models across representative workloads?
主な発見
- FlexGen achieves significantly higher throughput than state-of-the-art offloading systems on OPT-175B, reaching 1 token/s with an effective batch size of 144 on a single 16GB GPU.
- On HELM benchmark with a 30B model and a 16GB GPU, FlexGen can benchmark across 7 sub-scenarios in 21 hours.
- With 4-bit compression, FlexGen attains up to 100× higher maximum throughput (effective batch size 144) with 4000s latency by holding weights in CPU and removing disk offloading.
- Across OPT-6.7B and OPT-30B scales, FlexGen enables larger batch sizes and CPU/disk offloading to surpass baselines like DeepSpeed Zero-Inference and Hugging Face Accelerate in throughput.
- FlexGen’s 4-bit compression plus offloading yields substantial throughput improvements over baselines, illustrating the benefit of unified tensor placement and I/O-aware scheduling.
- Compared to Petals, FlexGen achieves higher per-GPU throughput and can have lower latency in certain scenarios.
この論文を読む
Nubintのビューアで原文を開き、読みながら気になる箇所をすぐに質問できます。
Nubintでこの論文を読むこのレビューはAIが論文の原文をもとに作成しました。引用する前に原文をご確認ください。