[Paper Review] Improving Large Models with Small models: Lower Costs and Better Performance

Chen Dong, Shuo Zhang+5 more|arXiv (Cornell University)|Jun 15, 2024

Scientific Computing and Data ManagementDecision Sciences3 citations

TL;DR

This paper proposes Data Shunt+ (DS+), a collaborative framework that improves large language models (LLMs) by offloading easy inputs to small models based on prediction confidence, reducing LLM inference costs by 68.82% while boosting accuracy from 94.43% (ChatGPT alone) to 95.64% on Amazon sentiment analysis. The method enables cost-effective, performance-enhanced inference through dynamic routing and knowledge distillation.

ABSTRACT

Pretrained large models (PLMs), such as ChatGPT, have demonstrated remarkable performance across diverse tasks. However, the significant computational requirements of PLMs have discouraged most product teams from running or fine-tuning them. In such cases, to harness the exceptional performance of PLMs, one must rely on expensive APIs, thereby exacerbating the economic burden. Despite the overall inferior performance of small models, in specific distributions, they can achieve comparable or even superior results. Consequently, some input can be processed exclusively by small models. On the other hand, certain tasks can be broken down into multiple subtasks, some of which can be completed without powerful capabilities. Under these circumstances, small models can handle the simple subtasks, allowing large models to focus on challenging subtasks, thus improving the performance. We propose Data Shunt$^+$ (DS$^+$), a general paradigm for collaboration of small and large models. DS$^+$ not only substantially reduces the cost associated with querying large models but also effectively improves large models' performance. For instance, ChatGPT achieves an accuracy of $94.43\%$ on Amazon Product sentiment analysis, and DS$^+$ achieves an accuracy of $95.64\%$, while the cost has been reduced to only $31.18\%$. Besides, experiments also prove that the proposed collaborative-based paradigm can better inject specific task knowledge into PLMs compared to fine-tuning.

Motivation & Objective

• To reduce the high inference cost of large pretrained language models (PLMs) in real-world product deployment.
• To improve the performance of large models by leveraging specialized small models for easy, distribution-conforming inputs.
• To enable efficient collaboration between small and large models without relying on expensive PLM APIs.
• To inject task-specific knowledge into PLMs more effectively than standard fine-tuning.

Proposed method

• DS+ uses a confidence threshold to route inputs: if a small model's prediction confidence exceeds a threshold δ, the input is processed solely by the small model; otherwise, it is sent to the large model.
• In the Small Model for Large Model (S4L) module, Prompt Pruning (PP) refines the large model’s prediction space by removing low-confidence classes from the prompt based on small model outputs.
• Prompt Transferring (PT) in S4L transfers knowledge from small models to large models by modifying the input prompt to emphasize regions where small models perform well.
• In the Large Model for Small Model (L4S) module, 2-Stage Confidence Distillation (2CD) transfers knowledge from large models to small models, preventing catastrophic forgetting during fine-tuning.
• The framework dynamically balances cost and performance by tuning the confidence threshold δ, with optimal values found in the range [0.97, 0.99] for high-performing small models.
• The method is generalizable across modalities and tasks, as demonstrated on text classification and other NLP benchmarks.

Experimental results

Research questions

• RQ1 Can routing inputs based on small model confidence improve large model performance while reducing inference cost?
• RQ2 How does confidence-based shunting compare to distribution-based or prediction-based routing in terms of accuracy and cost?
• RQ3 Can knowledge distillation from large models improve small model performance on hard samples without overfitting?
• RQ4 What is the optimal confidence threshold δ that balances cost reduction and performance gain in the DS+ framework?
• RQ5 Does the DS+ framework outperform standard fine-tuning in injecting task-specific knowledge into large models?

Key findings

• DS+ improved the accuracy of ChatGPT on Amazon Product sentiment analysis from 94.43% to 95.64%, demonstrating a 1.21% performance gain.
• The cost of querying the large model was reduced to only 31.18% of the original cost when using DS+, achieving a 68.82% reduction in inference cost.
• For small models with higher performance (e.g., fine-tuned BERT), the optimal confidence threshold δ was as low as 0.85, while for less accurate models (e.g., LSTM), δ needed to be above 0.97.
• When δ was increased from 0.97 to 0.99, the accuracy of DS+ with LSTM only improved slightly (from 94.20% to 94.45%), but the proportion of samples processed by the large model rose from 54.66% to 80.00%.
• Shunting based on confidence outperformed both distribution-based and prediction-based routing methods, with the latter showing only marginal gains and higher instability.
• The DS+ framework achieved better performance than the large model alone when using small models with high in-distribution accuracy, proving that collaboration enhances robustness and efficiency.