Skip to main content
QUICK REVIEW

[논문 리뷰] LLM in a flash: Efficient Large Language Model Inference with Limited Memory

Keivan Alizadeh, Iman Mirzadeh|arXiv (Cornell University)|2023. 12. 12.
Topic ModelingComputer Science인용 수 12
한 줄 요약

본 논문은 LLM 가중치를 플래시 메모리에 저장하고 창(windowing)과 행-열 번들링(row-column bundling) 같은 하드웨어 인식 기법을 사용해 필요 시 로드(on-demand)하도록 하여, DRAM보다 큰 모델의 추론을 가능하게 하고 순진한 로딩에 비해 큰 속도 향상을 달성한다.

ABSTRACT

Large language models (LLMs) are central to modern natural language processing, delivering exceptional performance in various tasks. However, their substantial computational and memory requirements present challenges, especially for devices with limited DRAM capacity. This paper tackles the challenge of efficiently running LLMs that exceed the available DRAM capacity by storing the model parameters in flash memory, but bringing them on demand to DRAM. Our method involves constructing an inference cost model that takes into account the characteristics of flash memory, guiding us to optimize in two critical areas: reducing the volume of data transferred from flash and reading data in larger, more contiguous chunks. Within this hardware-informed framework, we introduce two principal techniques. First, "windowing" strategically reduces data transfer by reusing previously activated neurons, and second, "row-column bundling", tailored to the sequential data access strengths of flash memory, increases the size of data chunks read from flash memory. These methods collectively enable running models up to twice the size of the available DRAM, with a 4-5x and 20-25x increase in inference speed compared to naive loading approaches in CPU and GPU, respectively. Our integration of sparsity awareness, context-adaptive loading, and a hardware-oriented design paves the way for effective inference of LLMs on devices with limited memory.

연구 동기 및 목표

  • Motivate running LLMs that exceed available DRAM by storing parameters in flash memory and loading on demand during inference.
  • Develop a hardware-informed cost model that accounts for flash and DRAM characteristics to guide data transfers.
  • Propose techniques to reduce data transfer and increase flash throughput to enable larger-on-device inference.
  • Show that sparsity and memory management strategies can significantly cut DRAM footprint and speed up inference.

제안 방법

  • Construct an inference cost model that incorporates flash memory, DRAM, and compute hardware characteristics.
  • Introduce windowing to reuse weights for past tokens and reduce IO requests.
  • Introduce row-column bundling to read larger contiguous chunks from flash by pairing related rows/columns.
  • Exploit FFN sparsity and a low-rank predictor to load only non-zero or likely-to-be-active weights.
  • Implement static DRAM preallocation and a sliding-window neuron data management scheme to minimize memory reallocation.
  • Evaluate with OPT 6.7B and sparsified Falcon 7B on CPU (DRAM+flash) and GPU setups, comparing against a full-load baseline.
Figure 1: Inference latency of 1 token when half the memory of the model is available. Our method selectively loads parameters on demand per token generation step. The latency is the time needed to load from flash multiple times back and forth during the generation of all tokens and the time needed
Figure 1: Inference latency of 1 token when half the memory of the model is available. Our method selectively loads parameters on demand per token generation step. The latency is the time needed to load from flash multiple times back and forth during the generation of all tokens and the time needed

실험 결과

연구 질문

  • RQ1Can on-demand loading from flash memory support LLM inference for models larger than available DRAM?
  • RQ2How can hardware-aware data transfer strategies (windowing and bundling) impact latency and throughput for flash-resident weights?
  • RQ3To what extent can sparsity prediction reduce data transfers without compromising accuracy on zero-shot tasks?
  • RQ4What are the practical latency and memory footprint trade-offs when running 7B–6.7B-scale models on edge-like hardware with limited DRAM?
  • RQ5How do these approaches compare to existing baselines/flexed offloading strategies (e.g., FlexGen) under similar memory constraints?

주요 결과

  • Models up to twice the size of available DRAM can be run by loading weights from flash on demand.
  • In CPU setups, speedups of 4–5x over naive loading; in GPU setups, speedups of 20–25x over naive loading.
  • Windowing reduces fresh data loading to about 2.4% of FFN neurons (with a 5-token window), saving DRAM bandwidth.
  • Row-column bundling doubles read chunk sizes and improves flash throughput by reading larger contiguous blocks.
  • A low-rank predictor and sparsity-aware loading preserve accuracy on zero-shot tasks while significantly reducing data transfers.
  • Compared against a full reload baseline, the proposed methods dramatically reduce I/O latency (e.g., from ~2330 ms to as low as ~87 ms per token in certain configurations).
(a) Bandwidth in a unified memory architecture
(a) Bandwidth in a unified memory architecture

이 논문 읽기

누빈트 뷰어에서 원문을 열고, 읽으면서 궁금한 곳을 바로 물어보세요.

누빈트에서 이 논문 읽기

이 리뷰는 AI가 논문 원문을 바탕으로 만들었습니다. 인용하기 전에 원문을 확인해 주세요.