[Paper Review] Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction

Haoran Qiu, Weichao Mao+8 more|arXiv (Cornell University)|Apr 12, 2024

Topic Modeling4 citations

TL;DR

This paper proposes SSJF, a speculative shortest-job-first scheduler for efficient interactive LLM serving that uses a lightweight proxy model to predict output sequence lengths, enabling better request scheduling. Evaluated on real-world datasets and production traces, SSJF reduces average job completion time by 30.5–39.6% and increases throughput by 2.2–3.6× across no batching, dynamic batching, and continuous batching settings without modifying memory or batching mechanisms.

ABSTRACT

Large language models (LLMs) have been driving a new wave of interactive AI applications across numerous domains. However, efficiently serving LLM inference requests is challenging due to their unpredictable execution times originating from the autoregressive nature of generative models. Existing LLM serving systems exploit first-come-first-serve (FCFS) scheduling, suffering from head-of-line blocking issues. To address the non-deterministic nature of LLMs and enable efficient interactive LLM serving, we present a speculative shortest-job-first (SSJF) scheduler that uses a light proxy model to predict LLM output sequence lengths. Our open-source SSJF implementation does not require changes to memory management or batching strategies. Evaluations on real-world datasets and production workload traces show that SSJF reduces average job completion times by 30.5-39.6% and increases throughput by 2.2-3.6x compared to FCFS schedulers, across no batching, dynamic batching, and continuous batching settings.

Motivation & Objective

• To address the non-deterministic execution times in autoregressive LLM inference caused by variable output sequence lengths.
• To reduce head-of-line blocking in LLM serving systems that rely on first-come-first-serve (FCFS) scheduling.
• To improve job completion time and system throughput in interactive LLM applications without altering memory management or batching strategies.
• To enable efficient scheduling across diverse batching settings: no batching, dynamic batching, and continuous batching.
• To demonstrate the feasibility and performance gains of using a lightweight proxy model for sequence length prediction in real-world LLM serving workloads.

Proposed method

• A small, fine-tuned BERT-base proxy model is trained to predict the output sequence length of LLM inference requests based on input queries.
• The predicted sequence length is used to estimate the total execution time, enabling speculative shortest-job-first (SSJF) scheduling.
• The SSJF scheduler prioritizes requests with shorter predicted execution times, reducing head-of-line blocking.
• The proxy model is applied independently of the main LLM inference pipeline, requiring no changes to memory or key-value cache management.
• The system supports multiple batching strategies: no batching, dynamic batching, and continuous (iteration) batching.
• The proxy model is retrained on real-world LLM conversation data (e.g., LMSYS-Chat-1M) to improve prediction accuracy for diverse input distributions.

Figure 1 . SSJF overview.

Experimental results

Research questions

• RQ1 Can a lightweight proxy model accurately predict the output sequence length of LLM inference requests?
• RQ2 Does using predicted sequence lengths to enable speculative shortest-job-first (SSJF) scheduling reduce average job completion time in LLM serving?
• RQ3 How does SSJF perform across different batching strategies, including no batching, dynamic batching, and continuous batching?
• RQ4 Can SSJF improve system throughput without modifying memory management or batching logic in existing LLM serving systems?
• RQ5 What are the performance gains of proxy model-based prediction compared to alternative approaches like self-prompting or regression on smaller datasets?

Key findings

• SSJF reduces average job completion time by 30.5–39.6% compared to FCFS scheduling across all evaluated batching settings.
• Throughput increases by 2.2–3.6× under varying request rates and burstiness levels when using SSJF instead of FCFS.
• The proxy model achieves high prediction accuracy on real-world LLM conversation data, with regression-based prediction outperforming multi-class classification.
• The method is effective across diverse batching strategies, including no batching, dynamic batching, and continuous batching, without requiring changes to memory or cache management.
• The proxy model-based approach outperforms self-prompting LLMs for length prediction, avoiding bias and reducing overhead.
• The open-source SSJF implementation demonstrates consistent performance gains on real production workload traces and popular open-source LLMs.

Figure 2 . Output length predictor architecture.