[Paper Review] Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models

Keisuke Kamahori, Tang, Tian+3 more|arXiv (Cornell University)|Feb 10, 2024

Gaussian Processes and Bayesian Inference4 citations

TL;DR

Fiddler proposes a CPU-GPU orchestration system for efficient inference of uncompressed Mixture-of-Experts (MoE) models in resource-constrained settings. By offloading expert computation to the CPU and minimizing GPU-CPU data movement, Fiddler achieves over 3 tokens per second on a 24GB GPU for the 90GB Mixtral-8x7B model, outperforming prior methods by up to 10.1x in latency.

ABSTRACT

Large Language Models (LLMs) with the Mixture-of-Experts (MoE) architectures have shown promising performance on various tasks. However, due to the huge model sizes, running them in resource-constrained environments where the GPU memory is not abundant is challenging. Some existing systems propose to use CPU resources to solve that, but they either suffer from the significant overhead of frequently moving data between CPU and GPU, or fail to consider distinct characteristics of CPUs and GPUs. This paper proposes Fiddler, a resource-efficient inference system for MoE models with limited GPU resources. Fiddler strategically utilizes CPU and GPU resources by determining the optimal execution strategy. Our evaluation shows that, unlike state-of-the-art systems that optimize for specific scenarios such as single batch inference or long prefill, Fiddler performs better in all scenarios. Compared against different baselines, Fiddler achieves 1.26 times speed up in single batch inference, 1.30 times in long prefill processing, and 11.57 times in beam search inference. The code of Fiddler is publicly available at https://github.com/efeslab/fiddler.

Motivation & Objective

• To enable efficient, low-latency inference of large, uncompressed Mixture-of-Experts (MoE) models in local, single-GPU settings with limited GPU memory.
• To address the high runtime overhead caused by frequent CPU-GPU data transfers in existing offloading systems for MoE models.
• To explore the feasibility of using CPU computation resources to reduce data movement and improve inference efficiency in sparse MoE architectures.
• To achieve high performance for single-batch, latency-sensitive inference without model quantization or multi-GPU setups.

Proposed method

• Fiddler dynamically assigns expert layers to either CPU or GPU based on a cost model that compares data transfer latency versus computation cost.
• It leverages CPU computation to execute expert layers instead of transferring weights from CPU to GPU memory, reducing PCIe bandwidth bottlenecks.
• The system uses a cost model to decide whether to run an expert on CPU or GPU, balancing computation and data movement overhead.
• Fiddler integrates with PyTorch and supports both prefill and decode stages with optimized memory management and execution scheduling.
• It employs a hybrid execution strategy where only a subset of experts are loaded on GPU, while the rest are computed on CPU.
• The system is designed for single-batch, local deployment scenarios, prioritizing low end-to-end latency.

Figure 1: Main idea of Fiddler . a) In existing offloading systems, expert weights are copied from the CPU to the GPU, and the computation happens on the GPU. b) In Fiddler , only the activations are copied between the CPU and GPU, and the computation happens on the CPU, drastically reducing the lat

Experimental results

Research questions

• RQ1 Can CPU computation be effectively utilized to reduce GPU-CPU data movement in MoE model inference?
• RQ2 How does the latency of CPU-based expert execution compare to the overhead of transferring expert weights over PCIe?
• RQ3 What is the performance gain of offloading expert computation to CPU instead of relying solely on GPU memory or data transfer?
• RQ4 How does Fiddler’s orchestration strategy scale across different hardware configurations with limited GPU memory?
• RQ5 Can Fiddler achieve high throughput for uncompressed MoE models without model quantization or multi-GPU setups?

Key findings

• Fiddler achieves over 3 tokens per second for the uncompressed Mixtral-8x7B model on a single 24GB GPU, enabling real-time inference in local settings.
• On the Quadro RTX 6000, Fiddler reduces single-batch inference latency by 8.2x compared to Eliseev & Mazur (2023) and 19.4x compared to DeepSpeed-MII.
• On the L4 GPU, Fiddler achieves 10.1x speedup over Eliseev & Mazur (2023) and 22.5x over DeepSpeed-MII in single-batch inference.
• The performance improvement is most significant for longer generation lengths, where prefill latency is amortized.
• Fiddler’s CPU-based expert execution strategy outperforms GPU data transfer even with lower CPU compute throughput, due to PCIe bandwidth limitations.
• The system enables deployment of 90GB+ models like Mixtral-8x7B on consumer-grade GPUs without quantization or multi-GPU setups.

Figure 2: The performance of Fiddler and baseline methods measured by the number of tokens generated per second, with 24 different configurations of input/output length. The rightmost set of bars shows the average of 24 configurations.