[Paper Review] FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance

Lingjiao Chen, Matei Zaharia+1 more|arXiv (Cornell University)|May 9, 2023

Topic ModelingComputer Science50 citations

TL;DR

FrugalGPT introduces LLM cascade to adaptively select among multiple LLM APIs to cut costs by up to 98% while matching or exceeding the best single LLM’s accuracy.

ABSTRACT

There is a rapidly growing number of large language models (LLMs) that users can query for a fee. We review the cost associated with querying popular LLM APIs, e.g. GPT-4, ChatGPT, J1-Jumbo, and find that these models have heterogeneous pricing structures, with fees that can differ by two orders of magnitude. In particular, using LLMs on large collections of queries and text can be expensive. Motivated by this, we outline and discuss three types of strategies that users can exploit to reduce the inference cost associated with using LLMs: 1) prompt adaptation, 2) LLM approximation, and 3) LLM cascade. As an example, we propose FrugalGPT, a simple yet flexible instantiation of LLM cascade which learns which combinations of LLMs to use for different queries in order to reduce cost and improve accuracy. Our experiments show that FrugalGPT can match the performance of the best individual LLM (e.g. GPT-4) with up to 98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost. The ideas and findings presented here lay a foundation for using LLMs sustainably and efficiently.

Motivation & Objective

• Motivate the high cost of querying diverse LLM APIs and the need for budget-aware usage.
• Propose three cost-reduction strategies: prompt adaptation, LLM approximation, and LLM cascade.
• Demonstrate a simple FrugalGPT implementation (LLM cascade) that triages queries to cheaper models while maintaining performance.

Proposed method

• Define a budget-constrained problem for querying K LLM APIs with per-query cost components.
• Describe three cost-reduction strategies: prompt adaptation (smaller prompts, prompt selection, query concatenation), LLM approximation (completion cache, fine-tuning cheap models), and LLM cascade (generation scoring + router to select among APIs).
• Instantiate FrugalGPT as an LLM cascade with a cascade length of 3 using ChatGPT, GPT-3, GPT-4, etc., and learn query-specific API sequences to maximize reward under cost constraints.
• Formulate a mixed-integer optimization for selecting API lists and thresholds, then provide a practical optimizer that prunes search space and interpolates the objective for efficiency.

Experimental results

Research questions

• RQ1 Can budget-constrained use of multiple LLM APIs achieve comparable or better task performance than the best single LLM?
• RQ2 What cost savings and accuracy trade-offs are achievable with an adaptive LLM cascade across diverse tasks?
• RQ3 How do prompt adaptation, LLM approximation, and cascade strategies interact and compound when forming a FrugalGPT system?
• RQ4 What are the practical limitations and necessary conditions for training and deploying an LLM cascade in real-world settings?

Key findings

• FrugalGPT can match the performance of the best individual LLM (e.g., GPT-4) with up to 98% cost reduction.
• FrugalGPT can improve accuracy over GPT-4 by up to 4% at the same cost on evaluated tasks.
• Across HEADLINES, OVERRULING, and COQA, FrugalGPT shows substantial cost savings (50%–98%) while maintaining or enhancing accuracy.
• The cascade leverages generation diversity, where cheaper models correctly answer cases that expensive models get wrong, enabling improved overall performance.
• In case studies, FrugalGPT reduces reliance on GPT-4 by sequentially querying GPT-J and GPT-J-like models when they provide reliable outputs.