[Paper Review] PowerInfer-2: Fast Large Language Model Inference on a Smartphone

Zhenliang Xue, Yixin Song+4 more|arXiv (Cornell University)|Jun 10, 2024

Topic Modeling4 citations

TL;DR

PowerInfer-2 is a framework that enables high-speed, on-device inference of large language models (up to 47B parameters) on smartphones by decomposing matrix computations into fine-grained neuron cluster operations, leveraging heterogeneous hardware (CPU, GPU, NPU) via a polymorphic neuron engine and optimizing I/O with segmented caching and pipelining. It achieves up to 29.2× speedup over state-of-the-art systems and is the first to run TurboSparse-Mixtral-47B at 11.68 tokens/second on a smartphone.

ABSTRACT

Large language models (LLMs) on smartphones enable real-time AI assistance and privacy-preserving, offline operation. However, resource constraints of smartphones limit current deployments to small language models (SLMs), significantly compromising their capabilities. This paper introduces PowerInfer-2, a smartphone-based framework that enables fast inference for LLMs exceeding the memory capacity. The key insight is decomposing matrix operations into neuron clusters as the basic processing unit, which enables flexible scheduling and efficient I/O-computation pipelining. PowerInfer-2 leverages this neuron-cluster-based design in both computation and storage. For computation, neuron clusters with dense activations are processed on NPU, while sparse clusters use CPU. The storage engine provides a fine-grained pipeline mechanism that coordinates cluster-level computation and I/O operations, enhanced by a segmented neuron cache to reduce I/O activities. PowerInfer-2 achieves up to a 27.8x speed increase compared to state-of-the-art frameworks. PowerInfer-2 is the first system to serve a 47B LLM on a smartphone, achieving 11.68 tokens/s. Notably, these performance improvements preserve model quality with negligible accuracy degradation.

Motivation & Objective

• To enable high-speed, on-device inference of large language models (LLMs) on smartphones, even when models exceed device memory capacity.
• To overcome I/O bottlenecks caused by limited storage bandwidth and single-command queue in smartphones during model weight retrieval.
• To fully exploit heterogeneous XPU resources (CPU, GPU, NPU) in smartphones for efficient LLM inference.
• To reduce memory usage for in-memory models while maintaining high inference speed.
• To support extremely large MoE models like TurboSparse-Mixtral-47B on mobile devices for the first time.

Proposed method

• Decomposes coarse-grained matrix computations in LLM inference into fine-grained neuron cluster operations tailored to smartphone hardware characteristics.
• Introduces a polymorphic neuron engine that dynamically adapts computation patterns: large clusters for prefill (NPU-optimized) and small clusters for decoding (CPU-optimized).
• Employs segmented neuron caching at the neuron cluster level with type-specific strategies to improve cache hit rates and reduce I/O overhead.
• Uses fine-grained, neuron-cluster-level pipelining to overlap I/O operations with computation, effectively concealing I/O latency.
• Leverages activation sparsity in LLMs to cache frequently used neurons in memory and offload less active ones to flash storage.
• Supports both out-of-memory and in-memory inference scenarios, with dynamic weight offloading and memory optimization.

(a)

Experimental results

Research questions

• RQ1 Can a framework achieve high-speed LLM inference on smartphones for models larger than device memory?
• RQ2 How can heterogeneous XPU resources (CPU, GPU, NPU) be effectively utilized to accelerate LLM inference on mobile devices?
• RQ3 To what extent can I/O overhead from flash storage be reduced or hidden during LLM inference on smartphones?
• RQ4 Can significant memory savings be achieved for in-memory models without sacrificing inference speed?
• RQ5 Is it feasible to run extremely large MoE models like TurboSparse-Mixtral-47B on smartphones with real-time performance?

Key findings

• PowerInfer-2 achieves up to 29.2× speedup over state-of-the-art frameworks on two smartphones during out-of-memory inference.
• It is the first system to support the TurboSparse-Mixtral-47B model on a smartphone, achieving a generation rate of 11.68 tokens per second.
• For in-memory models, PowerInfer-2 reduces memory usage by approximately 40% while maintaining performance comparable to llama.cpp and MLC-LLM.
• When 50% of FFN weights are offloaded, PowerInfer-2 saves ~1.5GB of memory on the TurboSparse-Mistral-7B model, whereas llama.cpp degrades by 20.8× and MLC-LLM fails to run.
• The average per-token neuron cache miss rate is 3.5%, but the P99 miss rate reaches 18.9%, indicating high variability due to activation similarity differences.
• Decoding latency for TurboSparse-Mixtral-47B is 99.76ms (mean), 140.56ms (P99), with significant variation driven by activation patterns and cache behavior.

(b)