[Paper Review] Llumnix: Dynamic Scheduling for Large Language Model Serving

Biao Sun, Ziming Huang+5 more|arXiv (Cornell University)|Jun 5, 2024

Natural Language Processing Techniques4 citations

TL;DR

Llumnix is a dynamic scheduling system for large language model (LLM) serving that enables runtime request rescheduling across multiple model instances using efficient live migration of in-memory states. By emulating OS-like context switching, it improves tail latency by an order of magnitude, accelerates high-priority requests by up to 1.5×, and achieves up to 36% cost savings compared to state-of-the-art systems.

ABSTRACT

Inference serving for large language models (LLMs) is the key to unleashing their potential in people's daily lives. However, efficient LLM serving remains challenging today because the requests are inherently heterogeneous and unpredictable in terms of resource and latency requirements, as a result of the diverse applications and the dynamic execution nature of LLMs. Existing systems are fundamentally limited in handling these characteristics and cause problems such as severe queuing delays, poor tail latencies, and SLO violations. We introduce Llumnix, an LLM serving system that reacts to such heterogeneous and unpredictable requests by runtime rescheduling across multiple model instances. Similar to context switching across CPU cores in modern operating systems, Llumnix reschedules requests to improve load balancing and isolation, mitigate resource fragmentation, and differentiate request priorities and SLOs. Llumnix implements the rescheduling with an efficient and scalable live migration mechanism for requests and their in-memory states, and exploits it in a dynamic scheduling policy that unifies the multiple rescheduling scenarios elegantly. Our evaluations show that Llumnix improves tail latencies by an order of magnitude, accelerates high-priority requests by up to 1.5x, and delivers up to 36% cost savings while achieving similar tail latencies, compared against state-of-the-art LLM serving systems. Llumnix is publicly available at https://github.com/AlibabaPAI/llumnix.

Motivation & Objective

• Address the challenges of workload heterogeneity and execution unpredictability in LLM serving, which lead to poor isolation, memory fragmentation, and SLO violations.
• Overcome the limitations of static scheduling and generic policies in existing LLM serving systems that fail to handle dynamic, multi-tenant workloads effectively.
• Enable performance isolation and efficient resource utilization by dynamically rescheduling requests across instances based on real-time workload characteristics.
• Differentiate between request priorities and SLOs to accelerate time-sensitive applications while maintaining cost efficiency.
• Unify diverse rescheduling scenarios—load balancing, fragmentation mitigation, and priority differentiation—into a single, scalable scheduling policy.

Proposed method

• Leverage live migration of in-memory states (e.g., KV cache) during ongoing inference to enable runtime rescheduling with negligible downtime.
• Design a unified dynamic scheduling policy that reacts to workload dynamics, including request size, memory consumption, and latency objectives.
• Implement a lightweight, GPU-optimized migration mechanism inspired by virtual machine live migration, minimizing migration overhead regardless of sequence length.
• Use runtime monitoring to detect memory contention and performance interference, triggering proactive rescheduling to maintain SLOs and balance load.
• Support both single-model and multi-model serving by extending the scheduling policy to handle model variants and different precision/size configurations.
• Integrate with existing LLM inference engines to provide backward compatibility while enhancing scheduling intelligence.

Experimental results

Research questions

• RQ1 How can LLM serving systems effectively manage unpredictable, heterogeneous workloads with varying memory and latency demands?
• RQ2 To what extent can live request migration improve load balancing and reduce tail latency in dynamic LLM inference environments?
• RQ3 Can a unified scheduling policy simultaneously address performance isolation, memory fragmentation, and request prioritization in LLM serving?
• RQ4 How does dynamic rescheduling impact cost efficiency and resource utilization in multi-instance LLM deployments?
• RQ5 What is the performance overhead of live migration for long-running autoregressive LLM requests, and can it be made negligible?

Key findings

• Llumnix reduces tail latency by up to an order of magnitude compared to state-of-the-art LLM serving systems.
• High-priority requests experience up to 1.5× speedup due to intelligent scheduling and prioritization.
• The system achieves up to 36% cost savings while maintaining similar tail latency levels, demonstrating improved resource efficiency.
• Live migration incurs negligible downtime regardless of sequence length, enabling seamless runtime rescheduling.
• The unified dynamic scheduling policy effectively reconciles conflicting goals of load balancing, fragmentation control, and performance isolation.
• Llumnix outperforms existing systems in handling unpredictable workloads, especially under bursty and heterogeneous request patterns.