[Paper Review] RouteLLM: Learning to Route LLMs with Preference Data

Isaac Ong, Amjad Almahairi+6 more|arXiv (Cornell University)|Jun 26, 2024

Data Mining Algorithms and Applications4 citations

TL;DR

RouteLLM proposes a router model that dynamically selects between a strong and weak LLM at inference time to optimize cost-performance trade-offs. Trained on human preference data and enhanced with data augmentation, the router achieves over 2x cost reduction on benchmarks like MT Bench and GSM8K while maintaining high response quality, and generalizes effectively to new model pairs without retraining.

ABSTRACT

Large language models (LLMs) exhibit impressive capabilities across a wide range of tasks, yet the choice of which model to use often involves a trade-off between performance and cost. More powerful models, though effective, come with higher expenses, while less capable models are more cost-effective. To address this dilemma, we propose several efficient router models that dynamically select between a stronger and a weaker LLM during inference, aiming to optimize the balance between cost and response quality. We develop a training framework for these routers leveraging human preference data and data augmentation techniques to enhance performance. Our evaluation on widely-recognized benchmarks shows that our approach significantly reduces costs-by over 2 times in certain cases-without compromising the quality of responses. Interestingly, our router models also demonstrate significant transfer learning capabilities, maintaining their performance even when the strong and weak models are changed at test time. This highlights the potential of these routers to provide a cost-effective yet high-performance solution for deploying LLMs.

Motivation & Objective

• To address the cost-quality trade-off in LLM deployment by dynamically routing queries to stronger or weaker models.
• To develop a router training framework that leverages human preference data and data augmentation for improved generalization.
• To achieve significant cost savings while maintaining high response quality across diverse benchmarks.
• To evaluate router performance on out-of-distribution benchmarks and demonstrate transferability across different model pairs.
• To open-source code and preference data for reproducibility and community use.

Proposed method

• The router model is trained using human preference data from Chatbot Arena to learn which LLM (strong or weak) is preferred for each input query.
• Data augmentation is applied by generating synthetic preference comparisons using an LLM judge or in-domain data to improve generalization.
• Multiple router architectures are evaluated, including BERT, causal LLMs, matrix factorization, and similarity-weighted ranking.
• The router predicts the optimal model for each query based on its input embedding and learned preference patterns.
• Performance is evaluated using metrics like Call-Performance Threshold (CPT) and Average Performance Gain Recovered (APGR).
• The framework supports zero-shot transfer by generalizing to new strong-weak model pairs without retraining.

Experimental results

Research questions

• RQ1 Can a router model trained on human preference data effectively route queries between a strong and weak LLM to reduce cost while maintaining quality?
• RQ2 How does data augmentation impact router performance across diverse benchmarks?
• RQ3 To what extent can router models generalize to new model pairs not seen during training?
• RQ4 What is the inference overhead and cost efficiency of deploying such a router in practice?
• RQ5 How do different router architectures compare in terms of performance and scalability?

Key findings

• The best-performing router achieves a 3.66x cost reduction on MT Bench while maintaining 95% of GPT-4's response quality.
• On MMLU, the router reduces cost by 1.41x while achieving 92% of GPT-4's performance.
• On GSM8K, the router saves 1.49x in cost while maintaining 87% of GPT-4's quality.
• The router generalizes effectively to new model pairs, requiring up to 30% fewer GPT-4 calls than random routing to reach 80% CPT.
• Data augmentation significantly improves router performance, especially when training data closely matches evaluation data in distribution.
• Router inference overhead is minimal, adding less than 0.4% to the total cost of GPT-4 generation, demonstrating practical deployability.