[Paper Review] Mixture of Experts with Mixture of Precisions for Tuning Quality of Service

HamidReza Imani, Abdolah Amirany+1 more|arXiv (Cornell University)|Jul 19, 2024

Engineering Technology and Methodologies4 citations

TL;DR

This paper proposes an adaptive serving framework for Mixture-of-Experts (MoE) models that dynamically mixes 16-bit and 4-bit quantization across CPU and GPU to tune throughput and model quality. By partially quantizing only the expert layers, the approach achieves up to 13.00 tokens/second throughput on an A100 GPU with only a 2.62–2.80 perplexity increase on WikiText2 under maximum quantization, enabling fine-grained trade-offs in resource-constrained, dynamic environments.

ABSTRACT

The increasing demand for deploying large Mixture-of-Experts (MoE) models in resource-constrained environments necessitates efficient approaches to address their high memory and computational requirements challenges. Moreover, given that tasks come in different user-defined constraints and the available resources change over time in multi-tenant environments, it is necessary to design an approach which provides a flexible configuration space. This paper presents an adaptive serving approach for the efficient deployment of MoE models, capitalizing on partial quantization of the experts. By dynamically determining the number of quantized experts and their distribution across CPU and GPU, our approach explores the Pareto frontier and offers a fine-grained range of configurations for tuning throughput and model quality. Our evaluation on an NVIDIA A100 GPU using a Mixtral 8x7B MoE model for three language modelling benchmarks demonstrates that the throughput of token generation can be adjusted from 0.63 to 13.00 token per second. This enhancement comes with a marginal perplexity increase of 3.81 to 4.00, 13.59 to 14.17, and 7.24 to 7.40 for WikiText2, PTB, and C4 datasets respectively under maximum quantization. These results highlight the practical applicability of our approach in dynamic and accuracy-sensitive applications where both memory usage and output quality are important.

Motivation & Objective

• To address the high memory and computational overhead of deploying large MoE models in resource-constrained, multi-tenant environments.
• To enable dynamic configuration of model throughput and output quality under fluctuating user constraints and available GPU memory.
• To reduce data movement bottlenecks between CPU and GPU by selectively quantizing only the expert layers.
• To provide a fine-grained, flexible configuration space that allows runtime adaptation to changing system conditions.

Proposed method

• The method employs partial quantization, where only the expert feed-forward layers are quantized to 4-bit or 16-bit precision, while non-expert layers remain in 16-bit.
• It dynamically determines the number of 4-bit experts and their distribution between CPU and GPU based on available memory and performance goals.
• The approach uses a hybrid execution model where experts are loaded on GPU when possible and offloaded to CPU when necessary, minimizing data movement.
• Throughput and perplexity are optimized by exploring the Pareto frontier of configurations, balancing memory footprint, computation, and model quality.
• The system leverages PyTorch’s tensor operations and supports mixed-precision inference with minimal runtime overhead.
• Model serving is adapted at inference time, allowing runtime reconfiguration without retraining.

Experimental results

Research questions

• RQ1 How can MoE models be efficiently served in dynamic, single-GPU environments with fluctuating resource availability and user constraints?
• RQ2 What is the impact of partial expert quantization (4-bit vs. 16-bit) on model perplexity and inference throughput?
• RQ3 Can a mixed-precision, mixed-location (CPU/GPU) expert deployment strategy achieve a fine-grained trade-off between model quality and throughput?
• RQ4 How does the configuration space of quantized experts affect the memory footprint and performance of MoE inference?

Key findings

• Throughput for token generation ranges from 0.63 to 13.00 tokens per second on an NVIDIA A100 GPU under varying memory constraints.
• Perplexity increases only marginally under maximum quantization: from 2.62 to 2.80 on WikiText2, 6.48 to 7.24 on PTB, and 3.24 to 3.53 on C4.
• The model size is reduced from 94.21 GB (full 16-bit) to as low as 23.55 GB (fully 4-bit quantized experts), with only a 2.62 perplexity increase on WikiText2.
• The adaptive approach achieves a 75% reduction in model size (from 94.21 GB to 23.55 GB) while maintaining a perplexity increase of less than 10% on all benchmarks.
• The method enables hyperbolic growth in throughput when both available GPU memory and the number of quantized experts increase, indicating effective mitigation of memory bottlenecks.
• The approach outperforms homogeneous quantization baselines by offering a superior trade-off between model size, throughput, and output quality.